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Logical  methods  of  problem  solving  which  take  into  consideration 
both  the  relative  cost  and  effectiveness  of  alternatives  have  been  ac¬ 
cepted  military  staff  procedure  for  many  years.  However,  the  rapid  ad¬ 
vances  and  increased  diversity  in  technology  since  World  War  II  have 
increased  both  the  range  and  complexity  of  alternatives  in  weapons  sys¬ 
tems  and  organizations  to  carry  out  the  military  policy  of  the  United 
States.  The  difficulty  of  selection  has  been  compounded  by  the  high 
costs  of  technological  developments  and  the  limitations  of  peacetime 
budgets.  To  assist  in  these  increasingly  difficult  choices,  develop¬ 
ment  of  more  useful  and  effective  tools  has  been  required. 

The  Research  Analyses  Corporation  is  engaged  in  a  series  of  studies 
aimed  at  the  continued  development  of  tools  and  teaching  aids  to  assist 
the  Army  in  performing  and  evaluating  studies  containing  the  analysis 
of  alternatives.  As  one  step  in  this  research,  we  have  prepared  a  ten¬ 
tative  guide  which  may  be  of  use  in  reviewing  and  evaluating  these 
analyses.  Those  who  are  relatively  familiar  with  cost-effectiveness 
analyses  may  find  the  Guide  useful  as  a  convenient  checklist.  To  the 
less  experienced  man  it  may  serve  as  an  elementary  text.  In  its  pre¬ 
paration,  we  drew  not  only  on  our  own  experience,  but  on  that  of  many 
scholars  and  soldiers.  We  particularly  wish  to  thank  Col.  John  Newman 
and  the  members  of  his  Systems  Analysis  Division  of  the  Office  of  the 
Chief  of  Staff,  US  Army,  for  their  assistance  and  patience  in  helping 
our  analysts  with  many  concepts  and  problems.  We  also  wish  to  thank 
Col.  James  Hayes  of  the  Office  of  the  Deputy  Assistant  Secretary  of 
Defense  (Systems  Analysis)  for  his  helpful  suggestions. 

In  addition  to  the  stated  authors,  many  members  of  the  Research 
Analysis  Corporation  contributed  to  the  preparation  of  this  Guide.  In 
particular,  the  Guide  benefited  from  the  contributions  and  suggestions 
made  by  Mr.  Arnold  Meltsner,  Dr.  Edward  Bernan,  and  Mr.  John  Phillips. 

The  document  was  reviewed  by  Dr,  George  Fettee,  Dr.  John  Hardt,  and  Mr. 
Roderick  Dennehy.  Their  comments  were  very  helpful.  The  responsibility 
for  the  contents  rests  with  the  stated  authors. 

Many  of  the  problems  raised  in  the  Guide  require  a  fuller  discussion 
than  appropriate  to  this  document.  We  are  in  the  process  of  developing 
more  complete  guides  to  significant  aspects  of  ccot-effectiveness  analyses 
for  those  who  will  have  the  need  to  work  with  them.  For  example,  one 
of  tne  guides  in  preparation  will  treat  the  role  of  the  threat  in  cost- 
effectiveness  analysis;  another  is  an  introduction  to  military  cost 
analysis . 

These  guides  need  to  be  improved,  particularly  by  the  insights  of 
those  who  will  be  using  them.  All  comments  and  suggestions  will  be 
gratefully  received  and  should  be  sent  directly  to  the  Research  Analysis 
Corporation,  Economics  and  Costing  Division,  McLean,  Virginia,  22101. 
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Robert  N.  Grosse,  Chief 
Economics  and  Costing  Division 
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Chapter  I 
GENERAL  BACKGROUND 


Introduction 


To  assist  in  the  review  of  studies  cont84ning  cost-effectiveness 
analyses,  a  series  of  key  questions  with  explanatory  notes  have  been  pre¬ 
pared  and  are  contained  in  the  next  chapter.  These  questions,  taken  to¬ 
gether,  will  not  necessarily  cover  all  aspects  of  all  cost-effectiveness 
analyses.  No  one  general  list  of  questions  can  do  that.  Rather,  the 
questions  are  designed  to  focus  the  attention  of  the  reviewer  on  selected 
aspects  to  assist  him  in  evaluating  the  analysis.  All  the  questions  are 
not  applicable  to  all  studies  and  they  are  not  necessarily  of  equal  im¬ 
portance  to  those  studies  where  they  do  apply.  The  reviewer  must  exer¬ 
cise  his  judgment  on  whether  the  questions  are  applicable  and  the  degree 
of  applicability  to  the  study  being  reviewed.  Those  questions  that  are 
considered  particularly  important  and  of  widest  application  have  been 
underlined,  and  are  also  listed  separately  for  convenience  in  the  back 
as  "SELECTED  QUESTIONS . "*  This  document  is  intended  only  as  a  guide 
and  not  as  a  full  and  comprehensive  treatment  of  all  aspects  of  cost- 
effectiveness  analysis. 

Questions  that  do  not  bear  on  military  co3t-effectiveness  analyses 
are  not  included  in  the  next  chapter.  Furthermore,  no  questions  are  ad¬ 
dressed  to  the  subject  of  the  intuitive  judgment  and  other  factors  used 
in  making  decisions  to  which  cost-effectiveness  analyses  contribute. 


Cost-Effectiveness  Analysis  and  the  Estimate  of  the  Situation 

Cost-effectiveness  analysis  is  a  method  for  studying  how  to  make 
the  best  of  several  choices.  By  cost-effectiveness  is  meant  the  rela¬ 
tion  of  the  resources  required  (cost)  to  achieve  a  certain  ability  to 
accomplish  an  objective  (effectiveness).  The  term  cost-effectiveness  is 
always  used  in  relation  to  the  effectiveness  of  alternative  systems, 
organizations,  or  activities*- 


♦Questions  1,  4,  12,  13,  14,  22,  23,  31,  37,  44,  46,  57, 


60,  66,  and  73* 
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Cost-effecti\ -u  .;s  analysis  i?  based  on  the  economic  concept  that 
all  military  decisions  involve  the  allocation  (best  use)  of  limited  re¬ 
sources  among  competing  requirements.  The  allocation  is  determined  by 
studying  hov  to  get  the  best  use  of  the  available  resources.  This  same 
concept  is  embodied  in  Army  decision  processes.  It  is  used  by  a  combat 
commander  when  he  determines  (estimate  of  the  situation)  the  allocation 
of  his  resources  (forces)  among  the  main  and  secondary  efforts  and  re¬ 
serves  in  the  offense  or  between  the  forward  and  reserve  forces  in  the 
defense.  A  G3  usee  che  same  concept,  in  preparing  his  recommendations 
for  reallocating,  among  the  elements  of  the  command,  the  ammunition 
available  supply  rate  announced  by  the  higher  headquarters.  The  company 
commander  goes  through  the  same  process  in  deciding  how  to  spend  his 
company  funds. 

Although  cost-effectiveness  analysis  and  the  estimate  of  the  situa¬ 
tion  ire  similar  in  concept,  they  differ  in  several  aspects.  The  purpose 
of  an  estimate  of  the  situation  is  to  arrive  at  a  recommended  course  of 
action.  It  is  usually  a  process  to  arrive  at  decisions  to  solve  "today's 
problems  today."  It  does  not  concern  itstlf,  in  a  realistic  sense,  with 
problems, operations,  or  systems  of  the  future,  even  chough  it  is  some¬ 
times  not  clear  where  the  problem  of  today  ends  and  the  problem  of  to¬ 
morrow  starts.  Because  it  deals  with  relatively  immediate  problems,  the 
formulation  of  possible  courses  of  action  in  an  estimate  of  the  situa¬ 
tion  is  severely  limited.  The  resources  (forces  and  weapons)  available 
to  the  commander  are  fixed  by  what  has  been  made  available  and  there  is 
no  real  flexibility  in  changing  their  composition  or  basic  organization. 
In  practice,  it  is  also  usually  difficult  to  obtain  additional  resources 
from  the  next  higher  commander. 

Another  severe  constraint  on  the  estimate  of  the  situation  is  the 
time  factor.  In."  omation  is  usually  incomplete  arid  the  time  available 
before  a  decision  is.  req-iired  often  does  not  permit  filling  in  gaps — 
even  if  it  were  possible.  Often  there  is  only  sufficient  time  to  analyze 
the  mission,  gather  staff  estimates,  formulate  a  few  possible  courses  of 
action  and  quickly  weigh  these  courses  of  action  against  the  enemy  cap¬ 
abilities  (or  difficulties  to  be  overcome)  and  with  each  other,  and  se¬ 
lect  a  course  of  action  basta  on  some  criteria--often  called  the  govern¬ 
ing  factors.  Time  usually  does  not  permit  testing  the  range  of  the  de¬ 
pendence  of  the  proposed  course  of  action  on  the  staff  estimates  and 
planning  assumptions. 
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Military  cost-effectiveness  analysis  is  not  a  decision  process  but 
an  aid  in  facilitating  decisions  that  must  be  made  nov  in  regard  to  de¬ 
velopment,  force  composition,  and  logistical  and  manpower  policy  problems 
in  order  to  be  prepared  for  wars  in  the  future.  The  analytical  tech¬ 
niques  employed  in  cost-effectiveness  analysis  are  required  to  supplement 
those  employed  in  the  estimate  of  the  situation  because,  as  we  look  In¬ 
to  the  future,  the  number  of  uncertainties  multiply.  These  uncertainties 
include  such  things  as  planning  factors,  the  enemy  and  his  reactions, 
the  strategic  concept,  technology,  chance,  and  even  the  national  objec¬ 
tives  which  can  be  expected  to  change  in  the  future  as  alliances  shift 
and  new  forces  in  the  world  develop.  Advances  in  technology  create  new 
opportunities  that  may  require  changes  in  organization  and  doctrine  as 
well  as  hardware.  All  these  uncertainties  lead  to  a  large  number  of 
variables  that  must  be  considered.  Some  of  these  variables  are  subject 
to  our  control,  some  to  the  enemy's  anu  others  to  nobody's  control.  But 
all  are  variables,  and  all  are  interdependent . 

The  increase  in  numbers  and  kinds  of  variables  associated  with 
problems  of  the  future  can  be  illustrated,  on  a  small  scale,  in  a  hypo¬ 
thetical  study  of  a  future  weapons  system  for  an  infantry  platoon — 
assuming  that  the  infantry  platoon  will  be  present  in  the  time  frame  under 
consideration.  The  variables  that  would  require  study  wculd  include  such 
parameters  as  alternative  weapons  systems  that  can  be  available  in  the 
time  frame  under  study,  composition  (mix)  of  kinds  of  weapons  withih  the 
total  system,  the  number  of  individual  weapons  within  each  mix  of  wea¬ 
pons,  levels  of  warfare,  expected  locales  cT  war,  and  effects  of  support¬ 
ing  weapons  of  higher  echelons.  If  each  of  these  parameters  takes  only 
three  alternative  values;  for  example,  levels  of  warfare  to  be  considered 
are  nuclear  warfare,  conventional  war,  and  one  type  of  stability  of  op¬ 
erations,  over  700  cases  result — and  all  significant  parameters  have  not 
been  listed.  If  the  number  of  candidate  weapons  systems  is  Increased 
from  three  to  six  there  are  over  1^00  cases  to  be  considered. 

It  is  in  this  environment  of  uncertainties  and  flexibility  in  use 
and  interchangeability  of  resources  (people,  dollars,  and  hardware) 
that  cost-effectiveness  analysis  is  a  useful  aid.  It  assists  in  provi¬ 
ding  increased  insight  into  the  problem  and  as  much  relevant  information 
as  possible  in  order  that  the  decision  maker  can  concentrate  on  those 
areas  where  judgment  must  be  applied,  particularly  in  consideration  of 
qualitative  aspects  and  consistency  with  higher  echelon  considerations. 

For  example,  a  hypothetical  force  composition  problem  where  flexi¬ 
bility  in  force  composition  is  possible,  it  has  been  determined  that  the 
force  must  have  a  capability  to  destroy  certain  kinds  of  targets  at  cer¬ 
tain  expected  ranges.  Two  possible  alternatives  are  artillery  and  tac¬ 
tical  air.  The  time  required  and  the  cost  to  destroy  these  targets  by 
use  of  each  alternative  can  be  calculated.  However,  the  importance  of 
having  a  capability  to  attack  these  targets  at  any  time  of  the  day or 
night,  regardless  of  weather  conditions,  is  a  matter  of  judgment.  This 
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judgment  can  be  better  made  when  the  cost-effectiveness  of  each  alter¬ 
native  is  known,  in  other  words,  the  price  to  be  paid  for  an  all-weather 
capability  stated  in  sufficient  detail  and  accuracy  to  be  useful  for 
planning . 

The  effort  to  provide  information  so  the  commander  can  better  exer¬ 
cise  his  judgment  is  also  found  in  the  estimate  of  the  situation  pro¬ 
cess.  For  example,  a  combat  commander  can  better  apply  his  judgment  to 
selection  among  possible  courses  of  actions  when  he  has  staff  estimates-- 
even  if  only  rough--of  the  number  of  casualties  he  will  suffer  and  the 
time  required  to  accomplish  the  mission  for  each  of  the  proposed  courses 
of  action.  However,  the  variables  that  a  staff  estimate  must  contend 
with  are  relatively  limited.  The  friendly  organization  is  fixed,  there 
is  only  one  enemy  in  only  one  area  and  the  options  open  to  the  enemy  are 
relatively  few.  For  example,  the  enemy  can  attack,  defend,  or  execute 
some  variation  of  a  withdrawal.  For  practical  purposes,  neither  the 
enemy  nor  the  friendly  force  can  introduce  new  weapons  systems  or  change 
their  fundamental  organization  or  doctrines  in  the  time  period  covered 
by  the  estimate  of  the  situation. 

The  basis  of  cost-effectiveness  analysis  is  that  there  are  alter¬ 
nate  ways  of  reaching  an  objective  and  each  alternative  requires  cer¬ 
tain  resources  and  produces  certain  results.  This  is  the  same  basis 
of  the  estimate  of  the  situation  which  studies  proposed  courses  of 
actions,  each  of  which  requires  certain  resources  (forces  and  supplies) 
and  produces  certain  expected  results  (time  to  take  the  objective, 
casualties  incurred).  A  cost-effectiveness  analysis  is  designed  to 
examine  systematically  and  relate  costs,  effectiveness,  and  risks  of 
alternative  ways  of  accomplishing  an  objective  and  designing  addi¬ 
tional  alternatives  (proposed  courses  of  action)  if  those  examined  are 
found  wanting.  It  is  an  analysis  of  the  cost  and  effectiveness  of  a 
system,  such  as  a  forward  area  air  defense  system  or  an  air  mobile  di¬ 
vision,  and  all  of  the  system  implications.  It  cam  be  considered  as 
a  kind  of  Consumers  Research  to  assist  in  getting  the  most  for  the 
resources  to  be  expended  and  not  as  a  search  for  the  cheapest  regard¬ 
less  of  effectiveness. 


A  major  methodological  difference  between  cost-effectiveness  analy¬ 
ses  and  other  military  studies  is  the  manner  in  which  the  results  are 
presented.  A  staff  study  or  a  staff  estimate,  like  a  cost-effective¬ 
ness  analysis,  considers  courses  of  action  (alternatives).  However, 
the  staff  estimate  and  staff  study  usually  embody  a  single  recommen¬ 
dation  with  the  other  alternatives  either  rarely  mentioned  or  not  as 
fully  discussed  as  the  recommended  course  of  action.  The  commander 
(decision  maker)  is  given  the  full  reasoning  behind  the  recommended 
course  of  action  which  is  frequently  presented  so  that  the  only  option 
open  is  a  "yes"  or  "no"  decision. 

In  a  cost-effectiveness  analysis,  the  significant  alternatives, 
the  available  facts,  the  reasoning  process  and  the  pertinent  considera¬ 
tions  pertaining  to  each  significant  alternative  are  presented.  All 
identifiable  assumptions  and  data  are  presented  so  that  their  validity 
can  be  questioned.  In  addition,  and  this  is  a  major  goal  of  a  cost- 
effectiveness  analysis,  the  dependence  of  the  results  of  the  analyses 
on  these  assumptions  and  data  are  tested. 

The  staff  estimate  and  staff  study  do  identify  major  assumptions. 
However,  an  implied  assumption  is  often  introduced  when  several  diff¬ 
erent  courses  of  action  are  open  and  a  decision  is  made  to  proceed  in 
one  direction.  Such  a  decision  is  then  accepted  as  a  known  quantity 
when,  in  reality,  it  really  is  an  assumption.  There  are  many  reasons 
for  such  assumptions,  but  frequently  the  result  of  the  study  or  esti¬ 
mate  is  not  tested  for  sensitivity  to  such  hidden  assumptions. 

Cost-effectiveness  analysis  places  great  emphasis  on  use  of  num¬ 
bers  and  calculations  in  any  effort  to  determine  quantitative  factors 
where  possible.  Of  course,  there  are  many  aspects  of  military  activi¬ 
ties  that  cannot  be  reduced  to  a  quantitative  factor.  There  is  now  no 
valid  way  of  assigning  a  number  to  morale,  the  psychological  effects  of 
a  certain  military  operation,  or  a  host  of  other  factors.  However,  it 
is  possible  to  calculate  the  number  of  155-mm  howitzer  rounds  and  total 
cost  required  to  destroy  a  certain  type  of  target.  The  impact  of  fac¬ 
tors  such  as  morale,  training,  reliability  of  allies  cannot  yet  be  cal¬ 
culated  and  are  now  matters  of  intuitive  judgment.  A  cost-effectiveness 
analysis  seeks  to  quantify  what  can  be  logically  calculated  so  that  the 
decision  maker  knows  the  extent  to  which  intuitive  judgment  must  be  used 
in  making  a  decision. 
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Essential  Elements 


The  essential  elements  of  a  cost-effectiveness  analysis  are: 

1.  Objective(s)  (functions  to  be  accomplished). 

2.  Alternatives  (feasible  ways  of  achieving  the 
desired  military  capability  or  accomplishing 
the  function). 

3.  Cost  of  resources  required  by  each  alternative. 

b.  A  set  of  mathematical  or  logical  relationships 
among  the  objectives,  alternatives,  environment 
and  resources  (models). 

5.  A  criterion  for  choosing  the  preferred  alternative. 


The  Objective 

The  determination  of  the  objective  is  often  complex.  In  order  to 
design  alternatives  properly,  the  problem  must  be  analysed  to  determine 
the  real  functional  need  underlying  the  requirements  for  certain  organi¬ 
zations  and  hardware  systems.  Thorough  examination  of  the  functional 
need  usually  brings  insight  into  the  problem  and  leads  to  generating 
alternatives  that  may  accomplish  the  desired  goal.  Close  examination 
of  objectives  stated  only  in  terms  of  specific  organizations  or  systems 
often  discloses  that  the  net  result  is  not  a  significantly  new  or  improved 
capability  but  a  relatively  minor  product  improvement.  This  does  not 
imply  that  product  improvements  are  not  needed  but  rather  that  a  full 
understanding  of  the  true  significance  of  what  is  being  proposed  for 
purchase  is  necessary.  For  example,  in  stating  a  requirement  for  an 
artillery  system  with  a  specified  minimum  range  capability,  the  real 
objective  may  be  a  capability  to  destroy  certain  kinds  of  targets  under 
certain  conditions.  By  examining  the  problem  from  the  functional  basis, 
the  planner  is  better  able  to  understand  the  problem.  This  insight  may 
lead  to  other  alternatives  that  should  be  studied.  The  examination  may 
show  that  the  proposed  new  artillery  system  is  only  one  alternative  to 
accomplishing  the  objective  and  that  another  alternative  may  be  preferable. 

There  are  practical  limits  on  the  definition  of  the  objective. 

Every  military  activity  is  part  of  a  larger  activity  amd  it  is  necessary 
to  draw  the  line  at  some  point.  However,  the  objective  should  not  be 
unduly  restricted  by  confusion  with  performance  characteristics  such  as 
speeds,  weights,  muzzle  velocities,  hit-kill  probabilities,  and  so  forth. 
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Alternatives 


i 


In  military  planning  there  is  rarely  only  one  exclusive  way  of 
achieving  a  given  objective.  Each  way  has  its  own  price  tag  of  time, 
men,  facilities,  materiel,  and  money.  Assume,  for  example,  that  the 
planning  problem — admittedly  over-simplified--is  to  design  a  new  type 
of  division  with  certain  capabilities.  In  satisfying  these  capabilities, 
the  TOE  designer  has  many  alternatives.  For  the  same  capability,  is  it 
better  to  have  more  mobility  (trucks,  aircraft,  and  other  vehicles)  and 
less  manpower,  or  perhaps  more  mortars  and  fewer  riflemen?  The  alter¬ 
natives  are  limited  only  by  creative  imagination  and  good  military  judg¬ 
ment.  By  exploring  alternative  ways  of  using  resources  it  is  often 
possible  to  discover  ways  of  achieving  an  objective  with  fewer  resources, 
or  accomplishing  more  with  the  same  resources.  All  feasible  and  signi¬ 
ficant  capabilities  to  accomplish  the  objective  should  be  considered, 
including  the  capabilities  of  the  Navy,  Air  Force,  and  Marines.  Preju¬ 
dices,  “party-line"  and  other  forms  of  preconceived  notions  should  be 
avoided  in  the  design  of  alternatives. 

Cost 


Determining  the  cost  of  each  alternative  is  based  on  incremental 
costs.  These  are  the  net  costs  of  adopting  the  alternative.  Such  costs 
are  determined  after  due  allowances  for  those  resources  already  paid  for 
regardless  of  whether  the  alternative  is  adopted,  and  would  be  available 
for  use  under  the  alternative  if  it  were  adopted.  In  determining  the 
cost  of  an  alternative  all  the  resource  implications  are  considered. 

The  alternative  is  treated  in  a  system  context.  For  example,  the  cost, 
admittedly  oversimplified,  of  adopting  a  new  radio  would  include  not 
only  the  cost  of  the  radio  and  its  development,  but  also  the  costs  of 
training  people  to  operate  it,  the  total  co3t  of  maintaining  the  radios, 
and  the  cost  of  the  additional  radios  required  for  maintenance  float, 
replacement,  combat  consumption,  and  so  forth. 

Costs  need  not  be  stated  in  precise  terms  down  to  the  last  dollar 
or  man.  However,  the  costs  must  be  accurate  enough  to  permit  evaluating 
the  military  worth  (effectiveness)  together  with  the  costs.  Like  every¬ 
thing  else,  this  rule  must  be  applied  with  discretion.  In  dealing  with 
systems  way  out  in  the  future  the  accuracy  of  the  cost  estimate,  whether 
it  is  an  absolute  figure  or  a  range,  probably  is  inverse  to  the  distance 
out  in  the  future.  Usually  cost  estimates  are  tested  by  sensitivity 
analysis.  These  are  repetitive  analyses  using  different  quantitative 
values  to  determine  if  the  results  are  sensitive  to  the  values  assigned. 
Such  analyses  give  the  decision  maker  a  better  under standing  of  how  much 
uncertainty  is  involved  if  there  are  significant  errors  in  the  cost  es¬ 
timates.  He  can  then  better  judge  if  the  investment  is  worth  the  pay¬ 
off  considering  the  uncertainties  involved. 
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Models 


Models  ore  used  in  cost-effectiveness  analysis  to  cope  with  the  host 
of  variables  that  are  inherent  in  problems  of  the  future.  A  model  is 
simply  certain  relationships  expressed  in  some  way  to  simulate  repl  or 
expected  conditions  in  order  to  foresee,  even  to  a  limited  extent,  the 
expected  outcome  of  a  course  of  action.  Models  assist  in  simplifying 
the  problem,  in  identifying  the  significant  components  and  interrela¬ 
tions,  in  determining  which  variables  are  especially  important  for  the 
decision  at  issue,  and  which  variables  can  be  suppressed.  In  this  manner, 
the  decision  process  can  be  more  precisely  focussed  on  those  areas  which 
require  a  judgment  decision. 

Models  range  from  simple  graphs  to  complex  equations  and  can  also 
take  the  form  of  a  wargame  or  field  maneuver.  The  estimate  of  the  situ¬ 
ation  and  staff  estimates  also  use  models.  The  comparison  of  proposed 
courses  of  action  against  enemy  capabilities  or  expected  difficulties 
and  the  cc  oarison  among  the  proposed  courses  of  action  represent  uses 
of  models  to  foresee  the  future  outcome  of  an  action. 

All  models,  simple  or  complex,  are  abstractions  c?  the  real  world 
and  their  validity  depends  on  the  proper  selection  of  assumptions  and 
the  correctness  of  the  relations  portrayed,  and  the  pertinence  of  the 
factors  included  in  the  model.  Two  aspects  of  model  building  are  par¬ 
ticularly  troublesome,  quantification  and  the  treatment  of  uncertainty, 
dome  variables  are  difficult  to  quantify,  such  as  the  continued  avail¬ 
ability  of  certain  support  from  an  ally.  This  difficulty  leads  either 
to  the  neglect  of  such  variables  by  ignoring  them  or  by  a  qualitative 
modification  of  a  solution  derived  from  the  treatment  of  other  variables 
that  have  been  properly  quantified.  Such  treatment  often  results  in 
the  difficult-to-quantify  variables  being  lost  within  all  the  other 
qualitative  considerations  that  must  be  weighed  when  the  time  comes  to 
recommend  action  on  the  basis  of  the  solution  from  the  model. 

Tne  influence  of  the  variable  that  cannot  be  quantified  and  all  un¬ 
certainties,  must  be  specifically  addressed  in  the  model  unless  it  can 
be  demonstrated  by  logic  or  analysis  that  they  are  trivial,  affect  all 
alternatives  roughly  the  same,  or  the  results  are  insensitive  to  them. 
Guessing  may  lead  to  disaster.  For  example,  if  there  is  uncertainty 
about  8  factors,  a  best  guess  might  be  made  on  each  of  them.  If  there 
is  a  60^  probability  that  each  best  guess  is  right,  then  the  probability 
that  all  guesses  are  right  is  less  than  2*.  Relying  on  best  guesses, 
in  this  case,  would  be  ignoring  al^.  the  outcomes  with  more  than  98$  prob¬ 
ability  of  occurring.  Uncertainties  and  the  problem  of  the  factors  that 
cannot  be  quantified  can  be  handled  through  various  techniques  such  as 
Monte  Carlo  sampling,  contingency  analysis,  (see  Glossary)  and  even  war¬ 
gaming  for  certain  purposes. 
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Models  that  portray  relations  incorrectly  also  lead  to  false  re¬ 
sults.  For  example,  seme  models  are  based  on  the  persistence  principle 
which  states  that  what  is  happening  or  has  happened  will  persist.  This 
type  of  model  is  dangerous  except  for  very  short-term  uses.  For  ex¬ 
ample,  it  is  wrong  to  assume  that  the  enemy  tactics  used  during  the 
Korean  war  will  continue  to  be  used  in  the  future  against  new  types  of 
equipment  and  tactics  that  may  be  introduced.  Seme  models  depend  on 
extrapolation  which  assumes  that  trends  will  continue  uninterrupted. 

Such  models  lend  themselves  readily  to  mathematical  treatment  but  are 
often  erroneous  because  of  failure  to  consider  what  is  called  the  Law  of  Di¬ 
minishing  Returns.  For  example,  a  machine  gun  can  fire  at  a  certain 
high  rate.  However,  this  high  rate  cannot  be  maintained  for  very  long 
(extrapolation)  because  the  barrel  would  soon  be  burned  out. 

Models  can  be  classified  irto  two  general  types — exact  (determinis¬ 
tic)  or  probabilistic.  An  exact  model  of  warfare,  of  course,  is  im¬ 
possible  in  peacetime.  However,  it  is  possible  to  create  an  almost 
exact  model  of  some  specific  piece  of  hardware  or  activity  and  subject 
it  to  test.  The  final  product  of  the  model  will  then  closely  approxi¬ 
mate  the  results  from  the  actual  hardware  or  activity.  March  graphs 
used  for  planning  administrative  movements  are  examples  of  deterministic 
models.  Most  military  problems  are,  by  nature,  made  up  of  uncertainties. 
Consequently,  they  are  considered  as  probabilistic  when  the  uncertainty 
is  identified  by  a  probability  factor.  For  example,  a  wargame  using  a 
certain  kill  probability  for  an  air  defense  system  is  a  probabilistic 
model . 

The  construction  of  models  to  evaluate  effectiveness  is  often  diffi¬ 
cult.  The  difficulties  arise  in  selection  of  criteria  of  effectiveness. 

It  is  relatively  easy  to  measure  the  comparative  effectiveness  of  two 
similar  pieces  of  equipment  designed  to  accomplish  the  same  general  ob¬ 
jective  as,  for  example,  in  comparison  of  a  towed  105-mm  and  a  self- 
propelled  105-mm  howitzer.  However,  it  is  more  difficult  to  compare  the 
effectiveness  of  general  purpose  force  organizations  such  as  two  diff¬ 
erent  kinds  of  divisions  or  even  two  equal- strength  infantry  battalions 
having  the  seme  general  kinds  of  weapons  but  one  having  three  rifle  com¬ 
panies  and  the  other  having  four.  The  impact  on  effectiveness  of  in¬ 
tangibles  such  as  morale  and  leadership  can  hardly  be  calculated  and 
requires  the  application  of  judgment.  Each  study  virtually  requires  a 
consideration  of  its  own  criteria  of  effectiveness. 

Models  used  in  cost-effectiveness  analysis  sometimes  tend  to  become 
mathematical  and  abstract.  Consequently,  they  may  be  difficult  to  under¬ 
stand.  A  good  cost-effectiveness  analysis  strikes  a  balance  in  the  use 
cf  models  between  simplicity  and  retention  of  enough  detail  to  ensure 
that  the  expected  outcome  of  an  expected  action  will  be  adequately  por¬ 
trayed.  In  any  case  all  models  have  certain  common  elements.  These 
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are  broadly  stated  as  a  definition  of  the  problem,  principal  factors  or 
constraints,  verification  and  the  decision  process-~or  application  of 
criteria. .  The  validity  of  conceptual  or  mathematical  models  cannot  be 
verified  in  a  cost-effectiveness  analysis  by  controlled  experiments. 

At  the  best,  they  can  be  tested  by  their  workability.  Questions  37  to 
44  in  the  next  chapter  are  designed  to  assist  a  review  to  test  the  work¬ 
ability  of  models  used  in  cost-effectiveness  analyses. 


Criteria 


The  most  widely  used  criteria  in  Army  studies  for  selecting  the  pre¬ 
ferred  alternative  are  usually  based  on  either  equal  cost  or  equal  ef¬ 
fectiveness  of  the  alternatives.  Another  method,  known  as  incremental 
effectiveness  at  incremental  cost,  is  used  in  special  cases.  In  the 
equal  cost  form  it  is  assumed  that  there  is  an  arbitrary  fixed  budget 
or  series  of  fixed  budgets,  and  the  analysis  determines  which  alterna¬ 
tive  gives  the  greatest  effectiveness  for  the  same  expenditures  of  re¬ 
sources.  In  the  equal  effectiveness  form,  a  specified  and  measurable 
military  effecciveness  (capability)  is  stated  and  the  analysis  is  tc 
determine  which  alternative  achieves  this  effectiveness  at  least  cost. 

The  incremental  effectiveness  at  incremental  cost  method  relates  the  in¬ 
crease  in  effectiveness  achieved  to  the  associated  increase  in  resources 
involved.  This  method  is  normally  used  only  as  a  last  resort  when  neither 
costs  nor  effectiveness  of  alternatives  can  be  made  equal,  e.g.,  when  a 
capability  based  on  a  new  technology  is  to  be  added  to  the  force  and  this 
new  capability  cannot  be  approximated  by  any  practicable  combination  of 
existing  materiel  and  men. 


Role  of  Judgment 

f-dgment  is  used  throughout  a  cost-effectiveness  analysis  in  the 
same  manner  as  in  the  making  of  an  estimate  of  the  situation  or  a  staff 
estimate.  Judgment  is  used  in  analyzing  the  oojective,  deciding  which 
alternatives  (courses  of  action)  to  consider,  which  factors  are  relevant 
and  the  interrelations  among  these  factors,  which  numerical  values  are 
to  be  used,  and  in  analyzing  and  interpreting  the  results  of  the  analysis. 
The  goal  of  a  cost-effectiveness  analysis  is  to  keep  all  judgments  in 
plain  view  and  to  make  clear  the  logic  used.  It  also  shows  the  sensi¬ 
tivity  of  the  results  to  the  significant  judgments  made.  The  depth  of 
a  cost-effectiveness  analysis  is  tempered  by  the  time  and  manpower  avail¬ 
able  and  the  importance  of  the  subject  matter.  A  cost-effectiveness 
analysis  requires  resources  and  it  must  serve  as  an  aid  to  the  making 
of  decisions  and  not  be  a  mere  intellectual  exercise. 
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Review  of  Studies 


There  are  probably  almost  as  many  different  ways  of  reviewing  a 
study  containing  cost-effectiveness  analysis  as  there  are  reviewers. 
Furthermore,  the  time  available  for  review  is  variable  and  studies  lack 
a  common  format.  It  is  suggested  that  the  points  listed  below  be  checked 
specifically  in  the  early  stages  of  a  review. 

a.  Statement  of  criteria  used  to  judge  effectiveness. 

b.  Statement  of  criterion  used  to  select  preferred 
alternative . 

c.  Use  of  incremental  costs. 

d.  Explanation  of  logic  of  models. 

e.  Presence  or  lack  of  analysis  of  sensitivity  of  the 
results  to  significant  data  and  assumptions. 

Without  these  elements  being  present,  the  study  will  probably  be 
of  poor  quality. 

Army— conducted  studies  containing  cost-effectiveness  analysis 
usually  do  not  have  a  uniform  organizational  pattern  but  many  generally 
follow  the  Staff  Study  format.  On  that  basis,  the  key  questions  in  the 
next  chapter  have  been  grouped  under  these  headings: 

Statement  of  the  Problem 

Assumptions 

Alternatives 

Documentation 

Cost 

Relationships  (Models) 

Effect:1  veness 

Criter  a 

Conditions  and  Recommendations 

The  grouping  under  the  above  headings  inevitably  leads  to  some  dup¬ 
lication  of  material,  particularly  on  the  use  of  analytical  tools  such 
as  sensitivity  and  contingency  analysis.  This  duplication  has  been  kept 
to  a  minimum  but  full  coverage  has  been  retained  under  each  heading  as 
a  convenience  to  the  reviewer  who  wishes  to  refer  to  a  specific  heading. 

The  Glossary  is  designed  to  give  a  non-technical  definition  of 
terms  frequently  used  in  cost-effectiveness  analyses. 
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The  annotated  Bibliography  ha*  been  designed  for  the  reviewer  who 
desires  to  read  further  into  the  methodology  of  cost-effectiveness 
analysis. 
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Chapter  II 
KEY  QUESTIONS* 
STATEMENT  OF  THE  PRCBIEM 


1.  IS  THE  PROBLEM  STATED  THE  REAL  PROBLEM? 

An  improper  statement  of  the  problem  often  results  in  either  study¬ 
ing  the  wrong  problem  or  precluding  consideration  of  worthy  alternatives. 
These  defects  are  usually  avoided  by  a  statement  of  the  problem  in  terms 
of  a  functional  need--the  job(s)  to  be  done--without  implying  how  it  is 
to  be  done.  A  statement  of  the  problem  in  terms  of  requirements  for 
kinds  of  forces,  systems,  or  performance  characteristics,  except  if  it 
is  a  follow-on  to  a  previously  approved  study  of  a  functional  need, 
should  be  critically  examined  to  ensure  that  the  wrong  problem  is  not 
being  studied  and  that  worthy  alternatives  are  not  automatically  excluded 
from  consideration.  For  example,  although  the  stated  problem  (no  previous 
study  of  functional  need)  may  be  to  select  a  rifle  to  meet  certain 
capabilities  (requirement  statement),  the  reed  problem  might  be  provid¬ 
ing  the  rifle  squad  with  adequate  firepower  to  accomplish  certain  func¬ 
tions  (functional  need).  In  such  a  case,  a  rifle  is  only  one  possible 
alternative. 

A  word  of  caution  is  in  order.  There  often  is  a  practical  limit 
on  the  depth  of  the  statement  of  the  functional  need  or  the  study  may 
become  unmanageable.  For  example,  in  the  case  cited  the  functioned  need 
could  be  conceivably  so  stated  that  tne  rifle  squad  itself  becomes  only 
one  alternative  to  solving  a  larger  problem.  To  avoid  this  difficulty, 
either  certain  broader  decisions  must  be  considered  as  made,  thereby 
narrowing  the  scope  of  the  study,  or  the  broader  study  under taken.  When 
the  former  approach  is  taken,  the  study  is  known  as  a  suboptimization. 

The  reviewer,  based  on  his  knowledge  and  judgment,  must  determine  if  the 
suboptimization  has  so  narrowed  the  scope  of  the  problem  that  the  real 
problem  has  been  missed  or  worthwhile  alternatives  excluded. 


2.  DOES  THE  STUDY  IDENTIFY  IMPLIED  SIGNIFICANT  COMPONENTS  OF  THE 
PROBLEM  THAT  MUST  BE  FULLY  TREATED  IN  THE  STUDY? 

Like  the  mission  statement  in  an  estimate  of  the  situation,  the 
problem  to  be  treated  in  a  cost-effectiveness  analysis  must  be  analyzed 
to  identify  all  functions  that  must  be  performed.  Some  of  these  implied 


*Those  questions  that  are  considered  particularly  important  and  of  widest 
application  have  been  underlined. 
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functions  are  often  not  apparent  at  first.  The  reviewer  should  watch 
for  implied  significant  component  parts  of  the  problem  that  are  neither 
identified  nor  treated  fully  in  the  study.  The  reviewer  s’  ould  also 
watch  for  other  problems  that  are  opened  up  or  revealed  by  the  study 
that  should  be  further  investigated. 


ASSUMPTIONS 


3.  ABE  ALL  ASSUMPTIONS  IDENTIFIED? 

The  reviewer  should  watch  for  assumptions  that  cure  not  identified 
as  such  because  assumptions  imply  a  limitation  or  a  judgment.  In  order 
to  evaluate  the  study  properly,  it  is  necessary  to  assess  the  impact  of 
the  limitations  and  the  validity  of  the  judgments  contained  in  all  the 
assumptions.  An  example  of  a  common  assumption  that  is  often  not  iden¬ 
tified  is  that  a  given  unit  operates  by  itself.  As  a  result,  in  mea¬ 
suring  the  effectiveness  of  a  division,  for  example,  inadequate  consider¬ 
ation  is  sometimes  given  to  the  support  the  division  receives  from  non- 
divisional  units  such  as  corps  artillery  or  tactical  air  units.  This 
failure  to  consider  non-divisional  support  may  lead  to  erroneous  conclu¬ 
sions  and  recommendations.  Another  frequently  hidden  assumption  is  that 
the  enemy's  doctrine  and  tactics  are  rigid  although  ours  are  flexible. 


4.  ABE  THE  ASSUMPTIONS  UNDULY  RESTRICTIVE? 

Assumptions  are  properly  used  to  narrow  the  scope  of  the  study  to 
manageable  proportions.  However,  the  assumptions  should  be  examined  to 
determine  whether  they  unduly  restrict  the  study  by  eliminating  possible 
significant  alternatives  or  by  narrowing  th*  scope  of  consideration  to 
the  point  that  the  conclusions  and  recccmendations  may  be  in  error. 


This  examination  may  be  required  throughout  the  review  of  the  study  and 
not  only  during  the  review  of  the  stated  assumptions. 

Assumptions  covering  the  subjects  listed  below  often  unduly  re¬ 
strict  the  scope  of  the  study  and  lead  to  questionable  conclusions  and 
recommendations . 

a. 

Non-availability  or  limited  availability  of 
support  from  other  services  (e.g.,  tactical  air 
support  or  MATS  effort). 

b. 

Locale  of  operations. 

c . 

Duration  and  intensity  of  operations. 

d. 

Enemy  organization,  operations ,  and  reactions 
to  our  decisions. 

e. 

Time  period  covered. 
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5.  DO  ANY  OF  THE  mjOR  ASSUMPTIONS  UNJUSTIFIABLY  TREAT  QUANTITATIVE 
UNCERTAINTIES  AS  FACTS? 

An  uncertainty  can  be  defined  as  the  lack  of  definitive  knowledge 
for  assigning  values  or  probabilities  to  factors  that  influence  decisions. 
Uncertainties  can  be  either  quantitative  (risks)  or  qualitative.  (See 
UNCERTAINTY  and  RISK  in  Glossary. )  Examples  of  quantitative  uncertain¬ 
ties  are  hit-kill  probabilities,  equipment  availability  rates,  ammuni¬ 
tion  expenditure  rates,  and  reliability  statements.  The  availability 
of  base  rights  in  a  foreign  country  at  some  future  time,  or  the  start  of 
aggression  by  the  potential  enemy  in  a  given  y*.ar  are  examples  of  a 
qualitative  unceitainty.  (See  next  question.) 

The  reviewer  should  be  alert  for  stated  and  implied  major  assump¬ 
tions  that  assign  fixed  values  to  quantitative  uncertainties  and  then 
treat  these  estimates  as  facts.  A  common  example  is  the  assumption  that 
&  proposed  weapon  system  will  have  a  certain  hit-kill  probability.  It 
is  often  better  to  handle  significant  uncertainties  by  sensitivity  an¬ 
alysis.  This  is  a  repetitive  analysis  using  different  quantitative 
values  to  determine  if  the  results  are  sensitive  to  the  values  assigned. 
When  significant  uncertainties  are  treated  as  facts  by  assumption,  the 
conclusions  and  recommendations  of  the  study  may  be  no  more  valid  than 
the  assumption  unless  it  can  be  demonstrated  that  the  conclusions  and 
recommendations  are  not  sensitive  to  plausible  errors  in  the  "facts." 

The  number  of  sensitivity  analyses  required,  and  feasible,  is  a 
matter  of  judgment.  There  are  limits  to  the  time  and  manpower  available 
for  a  given  study.  Sometimes  an  educated  guess,  considering  all  the 
circumstances,  will  suffice.  In  effect,  the  reviewer  must  judge  whether 
the  study  agency  has  performed  adequate  sensitivity  analyses  considering 
all  the  circumstances,  the  importance  of  the  subject,  and  whether  fur¬ 
ther  sensitivity  analysis  isay  significantly  affect  the  conclusions  and 
recommendations , 


6.  DO  ANY  CF  THE  MAJCR  ASSUMPTIONS  TREAT  QUALITATIVE  UNCERTAINTIES 
AS  FACTS? 

Major  qualitative  uncertainties  treated  as  assumptions  also  tend  to 
dictate  conclusions.  A  common  qualitative  uncertainty  that  may  dictate 
the  conclusions  concerns  th~  estimate  of  the  enemy.  Many  studies  are 
based  cm  intelligence  estimates,  or  target  arrays  prepared  or  approved 
by  the  Defense  Intelligence  Agency.  However,  these  estimates  are  some¬ 
times  assumeu  to  be  facts.  In  such  cases,  this  often  results  in  the 
en*  nay  being  considered  to  be  inflexible  and  no  allowances  are  made  for 
him  to  react  in  different  ways  to  our  operations  or  to  our  introduction 
of  new  capabilities.  When  it  is  not  definitely  known  how  we  will  operate 
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or  be  equipped  10  years  hence,  it  is  questionable  to  assume  that  the 
enemy  operations  and  equipment  in  the  future  can  be  predicted  with  cer¬ 
tainty. 

Other  qualitative  uncertainties,  stated  or  implied,  that  should  be 
treated  with  caution  are  those  associated  with  political  considerations. 
Examples  are  availability  of  base  rights,  assurance  of  overflight  per¬ 
mission,  and  composition  of  political  and  military  alliances  on  either 
side. 


Treatment  of  the  kinds  of  uncertainties  discussed  above  in  an 
analysis  is  not  simple,  but  the  effects  of  such  uncertainties  cm  the 
conclusions  should  not  be  neglected  in  a  study.  One  method  to  cope 
with  significant  uncertainties  of  this  kind  is  to  use  contingency  analy¬ 
sis.  This  involves  repetitive  analysis  with  different  qualitative  as¬ 
sumptions,  such  as  type  of  conflict  or  enemy  capabilities,  to  determine 
their  effects  for  comparison  with  the  results  of  the  initial  analysis. 
The  amount  of  contingency  analysis  required  has  to  be  a  matter  of  judg¬ 
ment,  as  discussed  in  the  previous  question. 


7.  ARE  THE  MAJOR  ASSUMPTIONS  REASONABLE? 

Major  assumptions  should  also  be  tested  to  determine  if  they  are 
reasonable.  This  test  can  be  facilitated  if  the  study  documents  or 
provides  some  explanation  of  why  each  assumption  was  made  so  that  the 
reasons  can  be  evaluated  by  the  reviewer.  A  useful  technique  for  re¬ 
viewers  is  to  try  to  think  of  other  major  assumptions  that  are  plausible. 
If  these  invalidate  the  conclusions  and  recommendations,  then  the  study 
is  questionable. 
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ALTERNATIVES 


8.  ARE  CURRENT  CAPABILITIES  ADEQUATELY  CONSIDERED  AMONG  THE  ALTERNATIVES? 

Current  capabilities  should  not  be  omitted  from  consideration  in 
construction  of  alternatives  except  for  valid  reasons  that  are  clearly 
stated.  Valid  reasons  may  include  failure  of  the  current  system  to 
accomplish  the  current  mission,  or  a  significant  degradation  of  capa¬ 
bility  relative  to  that  of  the  potential  enemy.  Consideration  of  cur¬ 
rent  capabilities  sometimes  helps  show  whether  the  proposed  new  system 
or  organization  is  an  improvement  that  is  worth  the  expenditure  of  new 
resources.  By  considering  current  capabilities,  much  of  whose  costs 
are  already  paid  for,  as  an  alternative,  the  study  can  show  the  diff¬ 
erence  in  effectiveness  and  costs  that  result  from  the  adoption  of  the 
proposed  new  system  or  organization.  (See  question  23.)  Current  cap¬ 
abilities  should  also  be  considered,  where  appropriate,  as  a  component 
of  an  alternative. 


9.  ARE  "TRADE-OFFS"  WITH  EXISTING  SYSTEMS  OR  ORGANIZATIONS  ADEQUATELY 
CONSIDERED  WITHIN  THE  ALTERNATIVES? 

Where  appropriate,  the  design  of  alternatives  should  consider 
"trade-offs"  with  existing  systems  or  organizations.  Possible  examples 
are:  (l)  in  studying  the  increased  use  of  Army  transport  aircraft  an 
alternative  might  include  reduction  in  other  means  of  transport;  (2)  in 
a  study  on  an  improved  fire  control  system  an  alternative  might  include 
a  reduction  in  ammunition  stockage. 


10.  ARE  THE  APPROPRIATE  CAPABILITIES  OF  THE  AIR  FORCE,  NAVY,  OR 
MARINE  CORPS  CONSIDERED  AMONG  THE  ALTERNATIVES? 

The  alternatives  should  consider  the  capabilities  of  Air  Force, 
Navy,  or  Marine  Corps  as  appropriate.  The  Army  usually  conducts  combat 
operations  with  the  support  of  one  or  more  of  the  other  Services  and 
the  other  Services  are  charged  by  law  with  furnishing  certain  support 
to  the  Army.  These  types  of  supports  are  listed  in  JCS  Publication  2 
(UNAAF).  For  example,  CONUS  air  defense  is  not  the  exclusive  respon¬ 
sibility  of  either  the  Air  Force  or  the  Army.  A  CONUS  air  defense 
problem  must  consider  Army  surface-to-air  missiles.  Air  Force  manned 
interceptors,  and  Air  Force  surface-to-air  missiles. 

Current  and  projected  capabilities  of  the  other  Services  can  be 
obtained  from  a  number  of  different  sources  including  the  Five-Year 
Force  Structure  and  Financial  Flan  maintained  by  each  Service.  The 
reviewer  should  bear  in  mind  that  functions  such  as  air  defense,  the 
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attack  of  surface  targets,  reconnaissance  in  the  vicinity  of  the  FEBA, 
and  transportation  within  a  theater  are  not  the  exclusive  responsi¬ 
bilities  of  the  Army. 


11.  ARE  MIXTURES  Or*  SYSTEMS  (ORGANIZATIONS)  CONSIDERED  AMONG  THE 
ALTERNATIVES? 

The  reviewer  should  watch  for  failure  to  consider  appropriate  al¬ 
ternatives  that  arc  based  on  mixtures  of  two  or  more  systems  (organi¬ 
zations)  tc  combine  the  best  features  of  each.  For  example,  in  com¬ 
paring  certain  transportation  systems  one  alternative  for  surface 
transportation  might  be  a  combination  of  truck,  rail  and  water  systems 
rather  than  only  a  truck  system.  In  the  same  manner,  the  study  of  a 
proposed  new  missile  system  might  consider  as  an  alternative  a  suitable 
combination  of  existing  missile  and  gun  systems  and  aerial  fire  support 
rather  than  only  an  existing  missile  system. 


12 .  ARE  ANY  FEASIBLE  AflD  SIGNIFICANT  ALTERNATIVES  (KITTED? 

A  major  contribution  that  a  reviewer  can  make  is  to  point  out  sig¬ 
nificant  and  feasible  alternatives  that  the  study  may  have  failed  to 
consider.  If  any  of  the  answers  to  the  previous  questions  on  "Alter¬ 
natives"  are  in  the  negative  then  it  is  possible  that  seme  feasible  and 
significant  alternatives  were  not  considered.  However,  the  reviewer 
must  exercise  judgment  before  criticizing  a  study  for  failure  to  con¬ 
sider  all  possible  alternatives.  There  are  practical  limits  on  the 
time  and  manpower  available  for  a  given  study.  The  relative  importance 
of  the  decision  on  the  subject  under  study  will  also  influence  the  num¬ 
ber  of  alternatives  examined.  The  reviewer  should  consider  these  as¬ 
pects  in  determining  whether  feasible  and  significant  alternatives  have 
been  omitted  to  the  detriment  of  arriving  at  sound  recommendat i ons . 

On  the  other  hand,  a  large  number  of  alternatives  may  only  indi¬ 
cate  that  minor  variations  have  been  considered  as  new  alternatives. 
Excessive  use  of  such  minor  variations  as  alternatives  often  beclouds 
significant  choices. 


DOCUMENTATION  AND  DATA 


13.  IS  THE  STUDY  ADEQUATELY  DOCUMENTED? 

A  key  element  of  systematic  analysis  is  sufficient  documentation 
of  methods  and  sources  so  that  with  the  same  material,  other  study 
groups  can  arrive  at  substantially  the  same  results.  Without  such  docu¬ 
mentation,  a  study  appeals  for  acceptance  solely  on  faith  in  the  author¬ 
ity  and  expertise  of  the  study  group  and  without  critical  examination 
of  the  sources  and  methods  used  to  arrive  at  the  recommendations. 

The  test  of  adequacy  can  be  applied  by  examining  the  models,  data, 
assumptions,  etc.,  to  determine  if  they  are  stated  in  such  a  way  that 
another  study  agency  could  trace  through  the  steps  of  the  study  and 
arrive  at  substantially  the  same  results  and  conclusions.  A  study  that 
is  net  adequately  documented  will  usually  fare  poorly  when  reviewed  by 
agencies  lacking  the  detailed  knowledge  of  the  problem  that  can  some¬ 
times  compensate  for  poor  documentation.  Inadequately  documented 
studies  may  require  only  slight  additions  to  be  properly  documented. 


14.  ARE  THE  FACTS  STATED  CORRECT? 

It  is  usually  neither  possible  nor  necessary  for  a  reviewer  to 
verify  all  the  factual  material  presented  in  a  study,  but  it  is  ad¬ 
visable  to  spot  check.  Particular  attention  should  be  paid,  where 
possible,  to  the  factual  material  on  which  conclusions  and  recommenda¬ 
tions  depend.  If  many  errors  are  involved  then  a  thjrough  verification 
of  the  facts  presented  may  be  in  order. 

In  reviewing  factual  material,  its  source  should  be  examined  crit¬ 
ically.  For  example,  frequent  use  is  made  of  data  contained  in  FM 
101-10,  " Organization,  Technical,  and  Logistical  Data"  and  similar 
publications.  The  data  contained  in  these  manuals  are  usually  averages 
of  historical  data  obtained  from  certain  kinds  cf  operations  in  specific 
theaters.  The  unquestioning  use  of  these  average  figures  may  lead  to 
erroneous  conclusions  because  the  use  of  an  average  hides  significant 
variations  that  exist  in  the  real  world.  A  tank  battalion  does  not 
always  cover  the  same  number  of  miles  each  day  even  over  the  same 
terrain.  Further,  the  data  contained  in  the  reference  manurls  may  not 
have  been  computed  for  the  purpose  required  in  the  study  and  consider¬ 
ations  important  to  the  study  may  not  be  included  in  the  calculations. 
For  example,  ammunition  expenditure  rates  contained  in  FM  101-10  are 
based  on  World  War  II  and  Korean  experience  and  organizations.  The 
use  of  these  rates  for  projected  operations  in  the  1965-70  time  frame 
would  be  questionable. 
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Projection  of  current  operational  experiences  into  future  time 
frames  should  also  he  examined  critically.  For  example,  a  study  used 
as  data  that  an  armed  helicopter's  missions  are  k%  escort,  casualty 
production,  and  the  remaining  missions  for  suppressive  fire.  These 
data  were  obtained  from  experience  in  Viet  Nam  operations.  This  un¬ 
questioning  projection  of  such  data  into  future  operations  in  other 
areas  fails  to  allow  for  possible  introduction  of  significantly  new  US 
and  enemy  tactics  and  may  result  in  conclusions  and  recommendations  on 
how  better  to  cope  with  the  last  war. 


15.  ARE  THE  FACTS  STATED  WITH  PROPER  QUALIFICATION? 

In  addition  to  checking  the. validity  of  the  factual  material,  it 
is  good  practice  to  check  the  factual  material  for  completeness.  Some 
material  may  be  factually  correct  in  isolation  but  may  take  on  a  diff¬ 
erent  significance  when  other  facts  are  added.  For  example,  it  is  true 
that  infantry  units  can  march  at  an  average  rate  of  2.5  miles  per  hour. 
However,  this  rate  is  valid  only  on  relatively  level  roads. 


16.  ARE  FINDINGS  AND  DATA  FROM  FIELD  EXERCISES  AND  FIE  ID  TESTS  USED? 

Field  exercises  and  field  tests  can  be  excellent  sources  for  ef¬ 
fectiveness  data.  When  used  in  a  study,  such  data  should  be  carefully 
examined.  The  reviewer  should  determine  whether  the  data  were  obtained 
by  measurements  or  by  judgment  of  individuals  and  if  similar  data  would 
likely  be  obtained  if  the  field  test  or  field  exercise  were  conducted 
again  by  another  agency  or  unit.  The  circumstances  surrounding  the 
field  exercise  or  field  test  should  be  reviewed,  where  possible,  to 
determine  if  any  artificialities  (there  are  always  some  in  any  peace¬ 
time  operation)  were  of  sufficient  influence  to  affect  the  results  of 
the  study  based  on  field  data.  Field  exercises  usually  have  many  para¬ 
meters  and  very  few  runs,  therefore  making  it  very  difficult  to  single 
out  cause  and  effect. 

Common  artificialities  that  may  significantly  affect  data  from 
field  exercises  and  field  tests  include: 

a.  Inability  to  assess  effectiveness  of  air  defense 
fires,  air-to- surface  fires,  and  ground-to-ground 
fires. 

b.  Lack  of  realistic  levels  of  support  from  the 
other  Services  or  other  supporting  units.  Often 
this  support  is  either  not  available  or  avail¬ 
able  in  abnormally  large  amounts. 


c.  Use  of  administrative  breaks  for  rests,  in¬ 
tensive  resupply,  and  maintenance  operations. 

d.  Unrealistic  maneuver  and  deployment  because 
of  restricted  maneuver  areas. 

e.  The  units  or  quantities  of  materiel  tested 
are  not  a  valid  sample  either  because  of  in¬ 
adequate  size  or  of  bias  in  composition. 

f.  Poor  or  inadequate  reporting  of  events  of  the 
exercise. 

g.  Effect  on  actions  of  participants  caused  by 
use  of  only  blank  ammunition. 


17.  ARE  THE  DATA  FROM  SUPPORTING  WARGAMES  VALID? 

Studies  sometimes  use  the  findings  of  war games  as  facts.  In  eval¬ 
uating  such  facts,  the  reviewer  should  bear  in  mind  the  nature  of  war- 
games.  Basically,  a  wargame  involves  a  hypothetical  situation  in  which 
two  opposing  sides  interact  in  accordance  with  a  set  of  more  or  less 
definite  rules.  In  all  forms  of  wargames,  the  play  is  determined  either 
by  mechanistic  rules  or  judgments  made  by  individuals  or  both.  These 
rules  and  judgments  are  based  on  assumed  situations  and  known  or  assumed 
facts  and  system  characteristics.  Well  planned  and  executed  wargames 
are  excellent  teaching  devices  and  provide  the  participants  with  good 
insights  into  the  problem  gamed.  Such  games,  if  well  documented, 
usually  provide  a  body  of  synthetic  data  which,  when  analyzed,  pro¬ 
vides  clues  to  problem  areas  that  need  further  investigation. 

In  determining  the  validity  of  the  findings  of  wargames,  the  re¬ 
viewer  should  judge  how  well  the  game  portrayed  reality  and  should 
satisfy  himself  on  the  validity  of  the  judgments  and  assumptions  used 
in  the  conduct  of  the  game.  The  study  should  lay  out  for  scrutiny  the 
major  judgments  and  assumptions  used  in  the  wargame.  It  is  recognized 
that  it  is  usually  not  possible  to  lay  out  all  judgments  and  assump¬ 
tions  used  in  the  wargame.  In  any  case,  the  reviewer  should  weigh  the 
dependence  of  the  conclusions  and  recommendations  on  the  findings  of 
the  wargame  and  consider  whether  other  competent  players  playing  the 
same  game  would  have  arrived  at  similar  results. 


18.  ARE  THE  PERFORMANCE  CHARACTERISTICS  VALID? 

Performance  characteristics  data  are  often  the  key  elements  in 
the  determination  of  the  effectiveness  of  a  system.  In  evaluating 
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the  validity  of  performance  characteristics,  the  source  of  the  data 
should  be  examined.  When  the  claimed  performance  characteristics  are 
essential  to  the  conclusions  and  recommendations  and  the  source  of  the 
data  is  not  clearly  stated,  additional  information  may  be  required 
from  the  agency  that  prepared  the  study.  This  may  not  be  necessary 
if  the  study  contains  a  sensitivity  analysis  of  a  reasonable  range  of 
values  for  the  performance  characteristics. 

Performance  characteristics  based  on  a  manuf acturer ' s  claims  are 
often  too  optimistic.  Performance  characteristics  derived  from  tests 
at  research  installations  also  require  examination.  Sometimes,  such 
performance  characteristics  sure  derived  under  controlled  conditions 
that  neglect  the  man-machine  relation  that  exists  under  field  condi¬ 
tion.  Even  performance  characteristics  derived  from  field  tests  should 
be  examined.  Such  tests  can,  at  times,  produce  misleading  results  due 
to  artificialities  caused  by  various  peacetime  restrictions  such  as 
safety  regulations  and  choice  of  test  areas. 

If  faced  with  questionable  performance  characteristics  that  are 
key  to  the  conclusions,  the  reviewer  should  consider:  (l)  performing 
a  sensitivity  analysis  himself  if  his  time  and  the  data  in  the  study 
permit;  (2)  requesting  validation  of  the  performance  characteristics 
and  sensitivity  analysis. 


19.  ABE  ANY  OF  THE  DATA  DERIVED  FROM  QUESTIONNAIRES? 

The  data  obtained  from  questionnaires  should  be  examined  to  de¬ 
termine  the  validity  of  the  questions,  the  adequacy  of  the  sample  and 
statistical  procedures,  and  the  expertness  of  *he  personnel  questioned. 
For  example,  one  study  cited  data  on  the  xrequency  of  kinds  of  missions 
expected  to  be  flown  by  Army  aircraft  in  a  conventional  war.  The  data 
were  based  on  a  questionnaire  completed  by  Army  aviators  at  one  Army 
post.  There  was  no  operational  experience  applicable  to  the  study  and 
an  educated  guess  or  subjective  judgment  was  in  order.  However,  in 
this  case,  the  judgment  of  those  who  order  Army  aviation  missions 
flown  (commanders,  operations  and  intelligence  officers)  should  have 
been  elicited  rather  than  the  judgment  only  of  those  who  execute  the 
missions . 


20.  ARE  GUESSES  AND  INTUITIVE  JUDGMENTS  IDENTIFIED? 

At  times  it  is  necessary  to  fill  in  data  gaps  with  educated 
guesses  and  intuitive  judgment.  These  educated  guesses  and  judgments 
should  be  identified  in  the  study  and  not  "swept  under  the  rug."  The 
reviewer  should  evaluate  these  judgments  and  weigh  their  impact  on 
the  conclusions  and  recommendations. 
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COST 


21.  IS  THE  COST  MODEL  IDENTIFIED? 

Every  cost-effectiveness  analysis  contains  a  cost  model.  A  cost 
model  generates  cost  estimates  by  application  of  cost  estimating  rela¬ 
tions  and  cost  factors  to  specified  physical  resources.  (For  a  further 
discussion  on  models  in  general  see  question  37*)  This  model  can  be  very 
complex  and  computer  assisted  or  it  may  consist  of  a  few  relatively  simple 
equations  readily  computed  by  hand.  The  study  should  sufficiently  iden¬ 
tify  the  cost  model  so  that  the  reviewer  can  determine  how  the  total  sys¬ 
tem  cost  estimates  were  derived  from  the  material  in  the  study.  If  the 
material  in  the  study  does  not  permit  the  reviewer  to  do  this,  then  addi¬ 
tional  information  is  required  fran  the  agency  that  prepared  the  study. 

The  cost  models  are  utilized  to  estimate  the  probable  economic  im¬ 
pact  on  the  Service  (or  Nation)  of  introducing  a  new  capability.  For 
planning,  these  costs  are  normally  stated  in  terms  of  research  and  de¬ 
velopment  costs,  investment  costs,  and  operating  costs.  Research  and  de¬ 
velopment  costs  include  those  costs  primarily  associated  with  the  devel¬ 
opment  of  a  new  capability  to  the  point  where  it  is  ready  for  operational 
use.  Investment  costs  are  those  costs  beyond  the  development  phase  to 
introduce  a  new  capability  into  operational  use.  Operating  costs  are  re¬ 
curring  costs  required  to  operate  and  maintain  the  capability. 


22.  ARE  THE  COST  ESTIMATES  RELEVANT? 

Cost  estimates  depend  on  the  problem  under  study  and  can  rarely  be 
obtained  from  books  containing  cost  data  although  cost  factors  and  cost 
estimating  relations  (CERs)  can  sometimes  be  found  in  such  books.  For 
example,  a  hypothetical  study  considers  as  an  alternative  a  new  kind  of 
light  infantry  division  which  has  been  designed  to  the  extent  of  an  out¬ 
line  TOE.  The  answer  to  the  seemingly  simple  question,  "What  is  the 
cost  of  this  new  division?"  depends  on  many  factors  including: 

Will  it  be  an  additional  division  to  those  already 
in  the  force  structure? 

Will  it  replace  an  existing  division?  If  so,  what 
kind? 

Where  will  it  be  stationed?  e.g.,  in  the  CONUS, 

Pacific,  Europe,  etc. 
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Will  it  have  new  Standard  A  equipment,  or  will 
existing  assets  of  Standard  B  type  equipment  be 
used? 

Are  there  any  existing  Army  units  whose  personnel, 
equipment,  and  facilities  can  be  used  by  the  new 
division? 

The  determination  of  which  costs  are  relevant  requires  considerable 
analysis  and  judgment.  It  is  not  possible  to  prepare  a  universal  list 
of  costs  that  are  always  relevant.  Ideally,  a  study  should  indicate  why 
certain  costs  were  considered  relevant  and  others  not.  The  questions 
that  follow  are  designed  to  help  the  reviewer  determine  whether  the  cost 
estimates  used  in  a  study  are  relevant. 


23.  ARE  INCREMENTAL  COSTS  CONSIDERED? 

Inherited  assets  are  those  resources  such  as  installations,  equip* 
ment,  and  trained  personnel  inherited  from  earlier  systems  which  are 
phasing  out  of  the  force  structure  and  are  usable  in  one  or  more  of  the 
alternatives  under  study.  The  costs  which  are  usually  pertinent  for 
planning  purposes  are  those  costs  yet  to  be  incurred.  For  example,  a 
study  considers  as  an  alternative  the  conversion  of  certain  artillery 
units  from  tube  to  missile  weapons.  In  determining  the  incremental  costs 
consideration  should  be  given  to  the  inherited  assets  of  trained  per¬ 
sonnel,  equipment,  and  facilities  that  are  or  can  readily  be  made  common 
to  both  units. 

Sunk  costs  are  those  costs  already  expended.  These  previously  in¬ 
curred  costs  are  normally  excluded  from  costs  presented  in  cost-effective¬ 
ness  analysis.  For  example,  a  study  considers  as  possible  alternatives 
weapons  systems  A,  B,  and  C,  each  with  an  associated  research  and  devel¬ 
opment  cost.  Only  alternative  A  is  already  under  development.  The  cost 
already  expended  on  Alternative  A  is  a  sunk  cost  and  the  research  and 
development  cost  of  Alternative  A  in  the  study  should  be  only  what  must 
yet  be  spent,  (to  complete  the  research  and  development  of  Alternative 


An  occasional  error  is  the  failure  to  consider  the  research,  devel¬ 
opment  and  investment  costs  of  existing  systems  as  sunk  costs.  For  ex¬ 
ample,  in  a  hypothetical  study  of  the  conversion  of  certain  artillery 
units  from  tube  to  missile  weapons,  one  of  the  alternatives  is  retention 
of  all  of  the  tube  weapons  units.  The  cost  of  that  alternative  would  not 
include  the  sunk  costs  represented  by  the  research  and  development  and 
investment  costs  already  expended  in  bringing  those  units  into  the  force 
structure. 
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Including  the  costs  of  Inherited  assets  and  other  sunk  costs  leads 
to  distorted  cost  estimates  with  consequent  effect  on  the  conclusions 
and  recommendations. 


24.  ARE  DIRECTLY  RELATED  SUPPORT  COSTS  INCLUDED? 

Cost  estimates  of  systems  or  organizations  should  include  the  pro- 
port.  jnate  cost  of  those  other  units  or  elements  required  in  direct  sup¬ 
port.  For  example ,  the  cost  estimate  of  HAWK  battalions  should  include 
the  costs  of  the  associated  HAWK  direct  and  general  support  detachments. 
In  the  same  manner,  the  cost  of  aviation  units  should  Include  a  direct 
share  of  the  cost  of  aviation  maintenance  units.  Failure  to  include 
directly  related  support  costs  may  result  in  misleading  cost  estimates 
of  alternatives. 


25.  ARE  COMBAT  CONSUMPTION,  REPLACEMENT/C ONSUMPTICN,  AND  MAINTENANCE 

FLOAT  COSTS  INCLUDED? 

Cost  estimates  for  the  major  equipment  items  should  include  not  only 
the  operational  equipment  assigned  to  organizations,  but  also  the  costs 
for  those  additional  items  required  for  initial  stockage  as  well  as  re¬ 
placement  items  over  the  period  in  which  the  system  is  to  be  in  use. 

(See  question  32.)  If  the  resource  implications  for  procuring  and  main¬ 
taining  authorized  maintenance  float,  replacement/consumption,  and  com¬ 
bat  consumption  stockage  are  excluded,  the  total  costs  of  the  system  al¬ 
ternatives  may  be  significantly  misleading.  (These  levels  of  stockage 
are,  of  course,  subject  to  logistics  guidance.)  For  example,  a  common 
error  is  to  include  only  the  cost  of  the  basic  load  of  ammunition  and 
to  neglect  the  cost  of  the  additional  ammunition  requirements  for  support 
of  the  weapon  system  or  organization.  The  total  ammunition  required,  to 
include  peacetime  training  requirements  and  expenditures  in  the  first 
part  of  a  war  until  wartime  production  becomes  available,  must  be  pur¬ 
chased  and  stocked  in  peacetime. 


26.  ARE  ALL  TRAINING  COSTS  INCLUDED? 

The  resource  implications  of  training  military  personnel  can  be 
significant.  Initial  training  costs  represent  the  resources  required 
for  the  training  of  personnel  necessary  for  introduction  of  the  alter¬ 
native  into  the  force  structure.  The  availability  of  fully-trained  per¬ 
sonnel,  as  well  as  the  number  of  personnel  requiring  complete  or  tran¬ 
sitional  training,  are  taken  into  consideration  in  determining  the  re¬ 
sources  required.  Annual  training  costs  represent  the  resource  implica¬ 
tions  for  training  replacements.  These  replacements  are  required  be¬ 
cause  of  normal  attrition. 
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Training  costs  usually  include  such  items  as:  (l)  procurement  of 
equipment  utilized  for  training  purposes;  (2)  construction  of  any  neces¬ 
sary  additional  facilities;  (3)  operation  and  maintenance  costs  of  the 
facilities;  (4)  the  pay  and  allowances  of  the  trainees.  For  example,  the 
cost  implications  of  ccomunications-electronic  equipment  utilized  for 
training  purposes  could  be  highly  significant. 


27-  ABE  CONSTRUCTION  COSTS  INCLUDED? 

The  costs  for  additional  installations  or  facilities  are  often  over¬ 
looked  yet  these  costs  can  be  important .  If  the  study  does  not  include 
any  construction  costs  and  does  not  state  how  the  facilities  were  ob¬ 
tained,  then  the  reviewer  must  either  satisfy  himself  that  no  construc¬ 
tion  is  required  or  take  necessary  steps  to  have  the  ttudy  corrected. 


28.  ARE  THE  COST  DATA  REASONABLY  ACCURATE? 

Although  it  is  not  usually  possible  for  a  reviewer  to  check  all  cost 
data  for  accuracy,  he  should  spot-check  and  examine  the  sources  of  the 
data. 


Cost  data  furnished  by  manufacturers  should  be  viewed  critically. 
Experience  has  shewn  that  such  data  sure  usually  understated,  particular¬ 
ly  for  advanced  systems.  Advanced  system  costs  stated  as  an  exact  figure 
rather  than  as  estimated  lower  and  upper  value s  are  particularly  suspect. 

The  basis  of  the  cost  data  for  advanced  systems  should  be  included 
in  the  study.  There  are  a  number  of  ways  for  arriving  at  such  estimates. 
One  commonly  accepted  method  relates  the  cost  data  for  the  components  of 
existing  analogous  systems  to  the  cost  of  the  advanced  system.  Unsup¬ 
ported  cost  data  are  suspect. 

Great  accuracy  in  cost  estimates  is  not  required  and  often  is  not 
feasible.  In  fact,  in  dealing  with  costs  of  advanced  systems  it  is 
usually  more  realistic  to  have  a  range  of  possible  costs  (upper  and 
lower  values)  rather  than  the  pseudo-accuracy  of  one  cost  figure  which 
assumes  no  uncertainties  in  arriving  at  that  figure. 


29.  ARE  COST  ASPECTS  OF  ALL  ALTERNATIVES  TREATED  IN  A  COMPARABLE  MANNER? 

Inconsistency  in  handling  the  cost  aspects  of  competing  alternatives 
prevents  an  objective  evaluation  of  their  comparative  or  relative  costs 
and  usually  leads  to  erroneous  conclusions.  It  is  not  always  possible 
to  use  the  same  cost  estimating  technique  for  calculating  a  cost  element 
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such  as  attrition  replacements.  This  is  often  the  case  in  studies  in¬ 
volving  alternative  systems  of  other  military  services.  For  example, 
one  service  may  calculate  aircraft  attrition  replacement  as  a  function 
of  an  activity  rate  (e.g.,  per  100,000  flying  hours)  while  smother  ser¬ 
vice  may  calculate  it  as  a  function  of  the  activity  inventory  (3  percent 
of  the  active  inventory  per  year ) .  The  reviewer  should  determine  that 
the  final  dollar  estimate  is  related  to  the  actual  resource  requirements 
for  the  alternative  and  that  computational  peculiarities  do  not  distort 
the  cost  results. 

Treating  alternatives  in  a  comparable  manner  must  not  be  carried  to 
the  point  that  costs  which  may  be  insignificant  in  one  alternative  sure 
therefore  not  considered  at  all  in  other  alternatives.  For  example, 
civilian  personnel  might  not  be  used  in  one  alternative  but  may  be  re¬ 
quired  by  another  alternative  in  significant  numbers.  To  exclude  this 
cost  could  distort  the  results. 


30.  ARB  THE  COST  ESTIMATING  RELATIONS  VALID? 

Cost  estimating  relations  may  be  crude  factors,  simple  extrapola¬ 
tion  of  recent  experience,  or  complex  equations  with  many  variables.  In 
all  cases,  the  purpose  of  a  cost  estimating  relation  is  to  translate  a 
specification  of  a  physical  resource  into  a  cost.  The  design  of  valid 
cost  estimating  relations  is  a  complex  subject  beyond  the  scope  of  this 
publication.  However,  several  common  errors  made  in  establishing  cost 
relations  are  discussed  below. 

Cost  estimating  relations  should  be  based  on  current  data  or  dis¬ 
torted  estimates  may  result.  For  example,  the  maintenance  cost  per 
flying  hour  for  an  Army  helicopter  has  decreased  significantly  over  the 
past  several  years  as  new  helicopters  have  been  introduced  into  the  force 
structure.  If  the  cost  estimating  relations  used  in  a  study  were  based 
on  information  for  early  Army  helicopters  (e.g.,  19^  through  195*+  data) 
the  maintenance  cost  per  flying  hour  for  a  present  system  as  well  as  for 
future  systems  alternatives  could  be  distorted. 

At  times  a  properly  constructed  cost  estimating  relation  may  be 
inapplicable.  If  the  system  alternatives  are  very  advanced  developments, 
the  cost  estimating  relations  based  on  the  current  technology  may  lead 
to  false  results.  For  example,  the  V/STOL  aircraft  concept  represents 
a  departure  from  aircraft  currently  in  production.  While  many  design 
characteristics  may  be  similar  to  present  aircraft,  there  may  be  a  num¬ 
ber  of  factors  which  could  increase  the  complexity  and  hence,  the  cost 
of  the  aircraft;  a  cost  estimating  relation  based  on  the  present  state- 
of-the-art  may  not  be  appropriate. 


31.  IS  AN  AMORTIZED  COST  USED? 


Amortized  costs  reduce  the  total  program  cost  of  the  system  to  an 
annual  cost  by  taking  the  total  operating  cost  of  the  program,  adding  to 
it  the  researcn  and  development  and  investment  costs,  and  reducing  the 
total  to  an  annual  basis  by  dividing  by  the  number  of  years  of  expected 
service  life  of  the  system.  The  same  general  procedure  may  be  utilized 
to  amortize  the  costs  per  month,  per  day,  per  sortie,  etc.  Ihis  approach 
disguises  the  differences  between  annual  operating  costs  resulting  from 
shifting  deployment  patterns  over  the  life  of  the  system  and  from  a 
varying  set  of  inherited  assets  over  time.  This  approach  makes  an  ar¬ 
bitrary  allocation  of  the  fixed  costs  of  a  system  over  time.  There  is 
no  basis  for  the  assumption  that  the  last  year  of  system  life  must  be 
charged  with  the  same  amount  of  R&D  cost  as  the  first  year.  The  first 
year  gets  the  newest  technology;  the  last  year,  obsolete  technology. 
Further,  an  amortized  cost  does  not  present  a  true  picture  of  the  total 
resource  implications.  If  the  system  is  to  be  in  the  force  structure 
for  say  10  years  the  amortized  annual  costs  may  lot*  relatively  small, 
yet  in  reality  there  may  be  relatively  large  dollar  costs.  It  is  the 
total  cost  of  the  alternatives  which  is  of  primary  concern. 

The  reviewer  should  attempt  to  convert  amortized  costs  into  total 
program  costs  and  use  such  costs  for  comparative  purposes.  If  this  can¬ 
not  be  done  readily  from  the  material  contained  in  the  study,  then  addi¬ 
tional  information  is  required  from  the  study  agency. 


32.  WERE  PEACETIME  OR  WARTIME  COSTS  INCLUDED? 

The  results  of  a  cost-effectiveness  study  may  be  very  sensitive  to 
the  use  made  of  peacetime  and  wartime  costs.  The  use  of  peacetime  costs 
only  may  indicate  that  System  A  is  preferred  while  the  same  study,  if 
wartime  costs  were  used,  may  have  concluded  that  System  B  is  preferable. 

Peacetime  costs  may  be  defined  as  the  costs  associated  with  devel¬ 
oping,  buying,  and  maintaining  a  capability  for  potential  war  during 
peacetime.  Such  costs  also  include  combat  consumption  stocks  (war  re¬ 
serves)  to  cover  the  neriod  from  the  beginning  of  a  war  until  wartime 
production  is  able  to  replace  battle  losses.  Wartime  costs  are  the 
costs  of  procurement  after  the  war  has  begun,  as  is  the  cost  of  replacing 
the  combat  consumption  stocks  if  the  war  terminated  during  the  useful 
life  of  the  system. 


In  the  case  of  general  purpose  forces  there  may  be  significant  pro¬ 
duction  of  weapons  and  expenditure  of  resources  after  a  limited  conflict 
begins  (as  in  the  Korean  Conflict  and  the  military  assistance  rendered 
to  South  Vietnam).  In  this  case,  wartime  costs  could  be  significant. 


However,  wartime  costs  are  difficult  to  determine  because  of  uncertainty 
regarding  the  duration  of  the  war,  loss  rates,  and  missions  undertaken. 

The  reviewer  should  be  guided  in  considering  the  proper  cost  ap¬ 
proach  (peacetime  or  wartime)  by  existing  policy  or  directive  from  the 
agency  directing  that  the  study  be  made. 


33.  WAS  A  WARTIME  ORDNANCE  COST  PER  MISSION  USED? 

The  use  of  a  wartime  ordnance  cost  per  missis.:  should  be  reviewed 
carefully.  Variations  on  this  approach  include  ordnance  cost  per  target 
killed,  per  casualty  and  per  sortie.  This  approach  is  usually  deficient 
because  of  failure  to  consider  all  the  costs  of  putting  into  place  and 
maintaining  a  capability  for  potential  war  throughout  the  projected  life 
of  the  system  in  the  active  force  structure.  Often,  this  approach  in¬ 
cludes  only  the  amnunition  costs  expended  during  a  brief  battle,  and 
neglects  the  bulk  of  the  significant  costs  associated  with  developing, 
buying  and  operating  the  system  in  peacetime. 


34.  WAS  AN  AMORTIZED  WARTIME-PEACETIME  COST  USED? 

In  this  approach  the  total  peacetime  cost  of  the  system  is  reduced 
to  an  annual  basis  as  explained  in  question  31.  To  this  amortized  peace¬ 
time  cost  is  added  the  estimated  annual  wartime  replacement /consumption 
costs.  No  distinction  is  made  between  wartime  and  peacetime  costs. 

This  approach  is  deficient  because:  (l)  it  assumes  the  war  will  continue 
over  the  entire  projected  service  life  of  the  system;  (2)  the  cost  re¬ 
sults  use  weighted  wartime  costs;  (3)  wartime  and  peacetime  costs  may 
not  be  commensurable;  and  (4)  it  does  not  present  a  true  picture  of  the 
total  resource  implications  as  discussed  in  question  31. 

Adding  amortized  costs  in  one  stream  to  another  annual  cost  stream 
infers  that  both  cost  streams  represent  the  same  total  time  duration. 

If  this  is  not  the  case,  then  the  two  cost  streams  should  not  be  added 
together  because  they  are  incommensurables .  Ad.ing  the  amortized  peace¬ 
time  costs  to  the  annual  wartime  cost  implicitly  assumes  that  the  war 
will  continue  over  the  entire  "service^Xfe1,r~of  the  system.  If  the 
peacetime  costs  had  been  amortized  to  a  per  day  or  per  mission  cost  in¬ 
stead  of  a  per  year  basis,  the  same  result  would  hold;  the  inferer.ee 
being  that  the  war  would  continue  over  the  entire  "service  life"  of  the 
system.  The  implied  assumption  that  the  war  would  last  for  the  "service 
life"  of  the  weapon  system  is  questionable. 

Costs  computed  by  this  method  are  we ighted  because  wartime  costs 
do  not  cover  the  same  length  of  time  as  peacetime  costs.  Such  weighted 


results  favor  the  shorter  time  period— the  wartime  costs.  It  is  only 
when  the  two  cost  streams  we  of  equal  length  that  the  costs  results  are 
not  distorted. 

To  assume  that  wartime  and  peacetime  costs  are  commensurable  may  be 
erroneous.  This  assumes  a  common  measure  between  the  values  of  resources 
procured  in  wartime  and  those  procured  in  peacetime.  During  wartime, 
the  cost  of  a  resource  may  be  quite  different  from  that  in  peacetime. 
Military  budget  constraints  during  peacetime  and  physical  resource  con¬ 
straints  during  wartime  may  produce  entirely  different  sets  of  costs 
for  the  same  military  resources.  As  a  greater  proportion  of  the  national 
budget  is  shifted  to  military  purposes  during  wartime,  the  scarcity  of 
dollars  for  military  resources  may  become  relatively  less  or  more  than 
during  peacetime.  Commensurability  between  wartime  and  peacetime  costs 
will  depend  upon  sucn  uncertainties  as  the  type  and  duration  of  war  and 
whether  a  war  will  actually  occur. 


35-  WAS  A  DO LIAR  COST  ASSIGNED  TO  THE  LOSS  OF  HUMAN  LIFE? 

Frequently,  a  study  will  assign  a  dollar  cost  to  a  human  casualty. 
The  loss  of  human  life  is  certainly  important  in  selection  among  alter¬ 
natives.  However,  the  value  of  a  human  life  is  incommensurable  with  the 
dollar  costs  associated  with  an  alternative.  It  is  better  to  treat 
human  losses  as  a  separate  measure  without  assigning  dollar  /alues.  Man¬ 
power  availability  in  both  peace  and  war  is  very  important,  but  this 
problem  cannot  be  properly  treated  only  in  terms  of  dollar  costs.  Men 
and  dollars  may  not  be  interchanged. 

36.  IS  THE  SENSITIVITY  OF  COST  ASSUMPTIONS  EXAMINED? 

In  comparing  costs  of  alternative  systems,  it  is  important  t o  de¬ 
termine  whether  the  results  are  independent  of  the  cost  assumptions. 

For  example,  would  ten  years  cf  peacetime  operations  as  opposed  to  five 
make  a  significant  difference  in  the  relative  costs  of  the  alternatives? 
Would  it  make  any  difference  if  the  procurement  levels  or  n”rd)er  of  units 
to  be  organized  changed?  The  study  should  make  clear  the  sensitivity 
of  the  cost  estimates  to  the  major  cost  assumptions.  If  the  study  fails 
to  do  this,  the  reviewer  should  attempt  to  determine  if  there  is  any 
such  significant  sensitivity  by  rough  calculation. 
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RELATIONSHIPS  (MODELS) 


37.  ARE  THE  MODELS  ADEQUATELY  IDENTIFIED  AND  EXPLAINED? 

The  conclusions  and  recommendations  of  a  cost-effectiveness  analysis 
cannot  he  evaluated  properly  unless  the  models  are  adequately  ident-i'ied 
and  explained.  Every  model  portrays  the  real  or  expected  world  by  ab¬ 
straction  and  simplification  in  order  to  predict  the  outcome  of  a  possible 
action  (see  Glossary).  Therefore,  the  explanation  of  the  model  should 
be  sufficient  to  provide  ready  understanding  of  which  aspects  of  the 
real  world  are  included  in  the  model,  which  aspects  have  been  omitted, 
and  the  underlying  assumptions  for  the  abstraction.  Basically,  a  good 
model  emphasizes  the  specific  areas  in  which  decisions  are  to  be  made 
by  removing  those  relatively  constant  elements  of  the  real  or  expected 
world  that  can  be  described  with  a  great  degree  of  certainty. 

The  study  should  contain  sufficient  explanation  to  permit  tracing 
the  operation  of  the  model  from  input  to  output.  The  detail  should  be 
sufficient  to  permit  calculation  of  new  results  from  different  input 
values  (sensitivity  analysis).  In  cases  where  a  model  is  machine- 
programmed,  sufficient  explanation  should  be  provided  for  following  the 
general  logic  of  the  program. 

38.  ARE  COST  AND  EFFECTIVENESS  LINKED  LOGICALLY? 

A  properly  structured  cost-effectiveness  analysis  contains  a  number 
of  models  that  link  effectiveness  and  cost  through  logical  interrelations. 
Usually  there  are  some  kinds  of  an  effectiveness  model,  '  system  and 
organization  model,  a  cost  model,  and  a  cost-effectiveness  model.  The 
exact  nature  and  number  of  these  models  will  vary  with  the  problem. 

The  study  should  provide  sufficient  information  and  explanation  for 
the  reviewer  to  follow  the  logic  by  which  the  models  relate  cost  and 
effectiveness . 

An  effectiveness  model  relates  measures  of  effectiveness  to  measures 
of  performance  in  an  operational  context.  For  example,  a  study  on  com¬ 
bat  vehicle  weapons  systems  used  as  a  measure  of  effectiveness  the 
probability  of  1,  2,  3,  • friendly  tanks  winning  an  engagement  with 
1,  2,  3,  ...  enemy  tanks  under  different  tactical  situations.  This  was 
related  to  performance  measures  such  as  muzzle  velocity,  warhead  speci¬ 
fics,  turret  slewrates,  turret  stability,  hull  characteristics,  rate  of 
fire,  target  acquisition  accuracy,  and  others,  under  various  tactical 
situations  and  rules  for  conduct  of  fire. 

A  system  and  organization  model  describes  the  physical  resources 
required  to  provide  the  performance  used  in  the  effectiveness  model. 
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For  example,  in  the  combat  vehicle  weapons  system  study  referred  to, 
this  included  the  physical  description  of  each  alternative;  the  complete 
vehicle,  ammunition,  armament,  fire  control,  communications,  TCE  unit 
description,  the  support  and  maintenance  requirements,  and  so  forth, 
consistent  with  the  planned  operational  concept. 

A  cost  model  relates  dollar  costs1  to  the  physical  resources  (and 
their  peacetime  activity  rates)  described  in  the  system/organization 
model.  The  cost  model  applies  cost  estimating  relations  and  factors. 

For  example,  the  same  study  used  the  total  future  cost  of  acquisition 
and  ownership  (R&D,  initial  investment,  annual  operating)  for  various 
quantities  of  systems.  Included  in  these  total  costs  were  not  only  the 
development  and  procurement  of  the  preferre-  item  but  also  such  addi¬ 
tional  costs  caused  by  training  of  personnel,  peacetime  ammunition  use, 
equipment  maintenance,  etc.  (See  question  22.) 

The  cost-effectiveness  model  finally  relates  the  costs  of  each  al¬ 
ternative  to  its  effectiveness  under  varying  assumptions.  Depending  on 
the  criterion,  the  model  may  compare  effectiveness  and  costs  of  alter¬ 
natives  at  equal  cost,  at  equal  effectiveness,  or  at  different  cost  and 
different  effectiveness.  (See  page  10.^  The  method  and  the 

techniques  used  to  achieve  this  cost  and  effectiveness  relation  should 
be  logical  and  explained.  For  example,  in  one  anti-tank  weapons  study 
the  equal  effectiveness  method  (winning  the  duel  -  all  pertinent  factors 
considered)  was  employed.  Effectiveness  was  related  to  cost  by  a  nu¬ 
merical  formula  for  calculation  of  cost  of  achieving  duel  success  at  a 
given  range  under  specified  conditions.  This  permitted  plotting  the 


The  graph  shows  the  cost  of  winning  a  "duel,'*  i.e.,  killing  the 
target  at  various  ranges.  (The  graph  portrayed  above  is  highly  simpli¬ 
fied.  In  the  actual  study  rather  than  a  simple  line,  a  band  was  used  to 
portray  the  variance  in  costs  for  winning  a  duel  at  a  given  range.  See 
Bibliography  Item  No.  1,  pages  13-17  for  a  more  complete  description.) 
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39.  DOBS  THE  MODEL  TREAT  THE  PROBLEM  IN  A  SYSTEM  CONTEXT? 

Most  military  systems  have  many  subsystems,  sub -sub systems  and  so 
forth.  Models  should  provide  for  the  proper  relations  among  subsystems 
so  that  the  full  implications  of  a  change  in  one  part  of  the  system  will 
be  reflected  in  the  rest  of  the  system.  For  example,  a  model  in  a  study 
of  an  airborne  surveillance  system  must  not  only  show  the  interrelations 
among  the  aircraft  (or  drones^  the  sensors  and  their  maintenance,  but 
also  the  interrelations  with  the  information  processing  functions  to  be 
performed  on  the  ground. 


40.  DOES  THE  MODEL  A^LCW  FOR  ENEMY  REACTION? 

It  normally  takes  several  years  to  implement  fully  a  decision  to 
deploy  a  new  system.  Therefore,  the  enemy  should  be  considered  to  have 
time  to  adjust  to  our  system  decisions.  A  major  aspect  of  the  effective¬ 
ness  of  our  system  is  the  degree  to  which  it  makes  such  adaptation  for 
the  enemy  either  technologically  difficult  or  economically  unattractive. 

For  example,  a  study  of  a  proposed  system  was  based  on  its  incor¬ 
poration  into  the  current  force  structure.  The  model  for  judging  the 
effectiveness  of  the  proposed  system  was  dominated  by  current  or  recent 
conflict  situations  (e.g.,  Vietnam,  Korea,  Europe).  In  using  the  model 
to  evaluate  the  effectiveness  of  the  future  system  only  in  the.  light  of 
these  current  or  recent  conflict  scenarios,  the  study  failed  to  consider 
the  steps  that  the  enemy  could  take  to  counter  the  proposed  system. 

(See  question  6.) 

41.  ARE  STRAIGHT  EXTRAPOLATIONS  USED  WITHOUT  PROOF? 

While  straight  extrapolations  (linear  relation)  often  do  apply  over 
limited  ranges  of  performance,  consumption,  or  similar  planning  figures 
based  on  averages  of  large  numbers,  they  rarely  apply  to  effectiveness 
or  cost  data. 

For  example,  the  relation  between  the  total  weight  of  rations  for 
one  infantry  division -month  and  the  weight  for  10  division -months  is  a 
straight  extrapolation.  The  relation  between  the  total  cost  of  the  first 
100  and  that  of  the  first  1,000  units  of  a  new  main  battle  tank  is  not 
linear  or  a  straight  extrapolation.  If  a  missile  system  has  10  missile?, 
costs  $1,000,000,  and  is  50)6  effective  (on  some  valid  measure),  then  a 
missile  system  with  20  missiles,  costing  $2,000,000,  will  not  be  100)6 
effective  but  (at  best)  75)6. 
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42.  ARE  DETERMINISTIC  AND  PROBABILISTIC  MODELS  USED  PROPERLY? 

A  deterministic  model  (see  Glossary)  uses  relations  of  the  type, 

"If  A  is  5»  then  B  is  always  8."  A  probabilistic  model  (see  Glossary) 
uses  relation1  of  the  type,  "If  A  is  5j  then  B  will  be  6-10  in  50)6  of 
the  cases,  4  or  5  in  2%  of  the  cases,  and  11  or  12  in  25^  of  the  cases." 

Cost-effectiveness  analyses  frequently  require  many  intermediate 
calculations  involving  data.  The  indiscriminate  use  of  specific  values 
often  creates  what  is  in  effect  a  deterministic  model.  In  reality,  the 
majority  of  the  coefficients  and  planning  factors  used  in  models  are 
only  averages  with  variances  and  different  degrees  of  confidence.  The 
reviewer  should  try  to  identify  the  probable  range  of  variance  about 
the  averages  that  are  used  as  inputs  and  have  at  least  an  intuitive 
feeling  about  the  confidence  of  the  numerical  results. 

Additionally,  the  reviewer  should  distinguish  those  cases  in  which 
a  probabilistic  model  is  needed  to  reflect  the  real  world  situation. 
Deterministic  models  are  usually  appropriate  (l)  when  the  planning  fac¬ 
tor  has  an  insignificant  variance,  such  as  weight  of  rations  per  day 
per  man  for  large  forces,  (2)  if  the  uncertain  factor  is  multiplied  by 
a  point  value,  such  as  cost  of  $8,000  to  $12,000  per  man  for  a  force  of 
20,000  men,  (3)  a  varying  factor  is  multiplied  by  a  linear  function,  such 
as  an  uncertain  flying  hour  rate  (e.g.,  2  to  6  hours  per  day)  multiplied 
by  a  flying  hour  cost  function  of  $20  a  day  plus  $4o  per  flying  hour. 

The  deterministic  technique  is  correct  in  these  three  cases  because  it 
will  give  the  same  most  probable  result  as  if  probabilistic  techniques 
had  been  applied.  Of  course,  there  may  still  be  a  problem  if  the  most 
probable  result  is  not  the  only  one  of  interest. 

Probabilistic  models  are  used  where  the  variables  in  the  problem 
may  assume,  at  any  given  time,  any  one  value  of  a  known  range  and  fre¬ 
quency  of  values,  as  opposed  to  deterministic  models  which  use  fixed  or 
average  values  all  the  Lime.  There  are  two  principal  types  of  probabil¬ 
istic  models:  static  models  using  probability  statements  instead  of  other 
values,  and  dynamic  (stochastic)  models  involving  change. 

Some  stochastic  models  use  random  numbers,  representing  change,  to 
select  values  from  frequency  distributions  for  a  given  problem.  For  ex¬ 
ample,  an  analysis  of  a  maintenance  support  organization  may  include  a 
model  which  represents  the  demands  for  maintenance  effort  placed  on  the 
support  organization.  Of  any  100  jobs  (demands),  20  will  require  1  man¬ 
hour,  30  will  require  2  manhours,  10  will  require  3  manhours,  15  will 
require  4  manhours,  5  will  require  5  manhours,  10  will  require  10  man¬ 
hours,  5  will  require  20  manhours,  2  will  require  30  manhours,  2  will 
require  4o  manhours,  and  1  will  require  80  manhours.  This  information 
is  arranges  into  a  cumulative  distribution  as  shown  below: 
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To  represent  requests  for  work,  a  two-digit  random  number,  say,  37 , 
is  drawn  (from  a  table  of  random  numbers  or  a  random  number  generator); 
the  corresponding  value  is  2  manhours.  The  next  random  number  is,  say, 
84,  and  the  value  is  10  manhours.  This  process  continues  at  some  rate 
(which  is  probabilistic)  and  the  requests  for  maintenance  are  arranged 
(queued)  in  the  order  of  simulated  requests:  2  manhours,  10  manhours, 

4  manhours,  and  so  forth.  Available  maintenance  men  would  be  assigned 
to  requests  under  various  rules,  e.g.,  1  man  to  jobs  less  than  4  hours, 

2  men  to  jobs  of  4  to  8  hours,  etc.  The  model  would  keep  track  of  the 
time  elapsed  between  generating  and  completing  a  request  for  maintenance. 
In  this  manner,  the  relation  of  numbers  of  maintenance  personnel  and 
delay  can  be  determined  for  various  assumptions  about  demand  for  main¬ 
tenance  effort. 

The  so-called  Monte-Carlo  model  described  above  requires,  however, 
a  sufficient  number  of  repetitions  to  obtain  adequate  information  about 
the  range  of  values  of  the  solution. 

A  static  model  using  probability  statements  may,  for  example,  apply 
in  a  study  on  aircraft  vulnerability.  The  probability  of  survival  for 
a  specified  time  is  given  by  the  product  resulting  from  the  multiplica¬ 
tion  of  the  following  probabilities: 

Probability  of  aircraft  being  detected 
Probability  of  aircraft  being  acquired  by  a  weapon, 
if  detected 

Probability  of  being  hit,  if  acquired  by  a  weapon 
Probability  of  kill,  if  hit. 

Probability  data  for  each  of  the  probabilities  listed  above  are 
derived  from  tests  and  experiments. 


43.  IS  A  ZERO-SUM  GAME  MODEL  USED  WHERE  IT  IS  NOT  APPLICABLE? 

A  zero-sum  two-person  game  is  a  conflict  in  which  there  are  two 
sides  and  the  gains  of  one  side  equal  the  losses  of  the  other.  Most 
conflict  situations  do  not  justify  the  use  of  this  type  of  model.  For 
example,  in  a  hypothetical  study,  the  effectiveness  of  alternative  US 
tank  systems  was  based  on  a  study  of  duels  between  US  tanks  and  enemy 
tanks.  Duel  results  were  based  on  the  losses  incurred  by  each  side. 
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An  enemy  loss  of  one  tank  was  equated  to  a  US  gain  of  one  tank.  The 
net  US  gain  was  used  to  determine  the  effectiveness  for  each  alternative. 

Our  gain  is  not  the  enemy's  loss.  The  situation  is  not  always 
symmetrical.  The  attacker  must  move,  the  defender  must  inhibit  move¬ 
ment.  Hence,  the  objective  of  a  US  tank  may  differ  from  the  objective 
of  the  enemy  tank.  In  fact,  other  alternative  concepts  might  inhibit 
enemy  tank  movement  more  effectively  than  would  a  US  tank  similar  to  an 
enemy  tank. 

44.  ARE  THE  MODELS  INTUITIVELY  ACCEPTABLE? 

Models  tend  to  become  mathematical  end  many  are  difficult  to  under¬ 
stand  even  in  their  broad  aspects.  Yet,  overly- simplified  models  tend 
to  become  superficial  by  limitation  in  choice  of  detail  and  omission  of 
important  variables.  The  objective  of  a  good  model  is  to  be  near  enough 
to  reality  so  that  the  model  outputs  can  be  used  to  predict  some  portions 
of  the  future  with  an  acceptable  degree  of  confidence. 

Models  can  be  tested  by  determining  if  they  represent  correctly 
known  facts  and  situations  not  considered  in  the  study.  Conversely,  if 
absurd  facts  and  situations  are  introduced  into  the  model,  comparable 
absurd  answers  should  be  produced  by  the  model.  If  the  reviewer  is 
aware  of  special  cases  in  which  there  is  same  indication  of  the  outcome, 
the  model  can  be  tested  to  determine  if  the  suits  are  in  general  agree¬ 

ment  with  the  indicated  outcome.  Another  \  that  can  be  applied,  at 
times,  is  to  vary  some  of  the  principal  parameters  and  determine  if  the 
model  produces  results  that  are  consistent  and  plausible. 


EFFECTIVENESS 


45.  ABE  THE  MEASURES  OF  EFFECTIVENESS  IDENTIFIED? 

The  study  should  clearly  identify  the  standards  or  measures  used 
for  evaluating  the  effectiveness  of  the  system  or  organization  under 
study.  If  not  explicitly  stated,  the  reviewer  should  attempt  to  iden¬ 
tify  these  measures  from  the  material  contained  in  the  study.  The  con¬ 
clusions  and  recommendations  cannot  be  properly  evaluated,  particularly 
when  the  study  is  based  on  equal  cost  alternatives,  without  prior  evalu¬ 
ation  of  the  measures  of  effectiveness. 


46.  IS  THE  EFFECTIVENESS  MEASURE  APPROPRIATE  TO  THE  FUNCTION  OR  MISSION? 


The  reviewer  should  satisfy  himself  that  the  measures  used  to  eval¬ 
uate  effectiveness  are  appropriate  to  the  function  or  mission  of  the 
system  or  organization  under  study.  Failure  to  use  meaningful  measures 
of  effectiveness  is  a  major  contributing  factor  to  unacceptable  studies. 
Examination  of  the  effectiveness  measures  requires  analysis  and  sound 
military  judgment.  The  example  below  illustrates  one  use  of  an  effec¬ 
tiveness  measure  that  was  not  appropriate. 

In  a  study  of  selected  infantry  and  artillery  weapons  systems,  the 
measure  of  effectiveness  was  a  division  firepower  score.  This  score  was 
the  sum  of  the  firepower  scores  of  the  units  within  the  division.  The 
firepower  score  of  a  unit  was  based  on  sustained  rates  of  fire,  effective 
width  of  burst,  and  the  fragmentation  area  of  the  weapon  in  comparison 
with  other  weapons.  Specifically,  direct-fire  weapons  such  as  rifles 
were  assessed  in  terras  of  probable  hits  per  minute  against  personnel  in 
the  open.  Mortars  and  artillery  were  assessed  in  terms  of  maximum 
effective  range  and  lethal  area  coverage  per  minute. 

This  use  of  a  firepower  score  was  wrong  for  a  number  of  reasons. 
Primarily,  it  failed  to  differentiate  between  the  effectiveness  of  wea¬ 
pons  when  used  for  neutralization  and  when  used  to  produce  casualties. 

For  neutralization,  the  effectiveness  is  strongly  dependent  on  burst 
rate  of  fire,  incipient  damage  area  produced  by  the  burst,  and  ability 
to  maintain  fire  over  the  required  time  (the  latter  a  function  of  weapon 
characteristics  and  ammunition  requirements).  On  the  other  hand,  casual¬ 
ty  production  depends  strongly  on  the  probability  of  hit,  which  in  turn 
depends  on  target  acquisition  and  weapon  guidance  or  accuracy.  Thus, 
in  this  case,  several  measures  must  be  used  to  have  a  valid  analysis. 

The  total  division  firepower  score  used  in  the  study  also  assumed 
an  inexhaustible  and  uniform  supply  of  ammunition  regardless  of  whether 
the  weapon  was  a  rifle  company  machine  gun  or  a  division  general  support 
artillery  cannon. 
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47.  DO  THE  EFFECTIVENESS  MEASURES  IGNORE  SOKE  OBJECTIVES  AND  CONCENTRATE 

ATTENTION  ON  A  SINGLE  ONE? 

In  the  measurement  of  effectiveness,  the  reviewer  should  watch  for 
any  tendencies  to  concentrate  chi  only  one  or  two  objectives.  Such  a 
situation  indicates  an  unstated  assumption  that  other  objectives  are 
unimportant.  The  resulting  conclusions  and  recommendations,  if  imple¬ 
mented,  may  cause  an  imbalance  and  reduce  the  capability  to  achieve 
other  objectives. 

For  example,  a  study  indicates  that  the  most  vulnerable  element  in 
a  line  of  communications  system  are  the  bridges  in  a  rail  network  and 
measures  effectiveness  of  deployment  of  given  air  defense  units  by  degree 
of  protection  afforded  railway  bridges.  In  evaluating  the  overall  ef¬ 
fectiveness  of  the  air  defense  deployment,  the  study  fails  to  consider 
that  the  vulnerability  of  other  elements  in  the  line  of  communications 
system  may  be  greatly  increased  by  the  redeployment  of  the  air  defense. 

A  possible  test  for  effectiveness  measures  suspected  of  concentra¬ 
ting  on  a  single  objective  is  to  evaluate  them  against  a  hypothetical 
obviously  absurd  weapon  or  device  that  does  only  one  job.  Valid  mea¬ 
sures  of  effectiveness  should  show  an  absurd  hypothetical  weapon  or  de¬ 
vice  in  its  true  light. 


48.  ARE  PERFORMANCE  MEASURES  MISTAKEN  FOR  EFFECTIVENESS  MEASURES? 

Measures  of  performance  characteristics  are  sometimes  misconstrued 
as  measures  of  the  ability  of  the  system  or  organization  to  accomplish 
its  function.  Performance  characteristics  may  contribute  one  of  the 
many  inputs  required  to  achieve  the  effectiveness  of  the  system  or  or¬ 
ganization  as  a  whole.  For  example,  the  speed  of  movement  or  mobility 
of  a  unit  is  only  one  aspect  of  the  unit's  capability  to  accomplish  its 
function.  The  speed  at  which  a  unit  can  attack  the  enemy  is  not  in  it¬ 
self  a  measure  of  the  ability  of  the  unit  to  defeat  the  enemy.  The 
weapon  with  the  smaller  CEP  is  not  necessarily  the  more  effective  wea¬ 
pon;  the  relation  of  lethal  radius  to  CEP  may  be  more  significant.  Other 
factors  that  must  be  also  considered  in  weapon  effectiveness  include 
target  acquisition,  weapon  guidance,  and  target  size. 


49.  IS  THE  EFFECTIVENESS  CALCULATED  CN  THE  BASIS  OF  EITHER  A  COOPERATIVE 
ENEMY  OR  AN  OMNIPOTENT  ENEMY? 

Neither  basis  is  valid.  The  enemy  should  be  expected  to  adjust 
his  decisions  to  our  own  planning  as  much  as  his  resources  permit.  An 
unstated  assumption  that  the  enemy  is  inflexible  in  the  face  of  our 
changes  is  a  common  error  in  cost-effectiveness  studies. 
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Far  example ,  a  counter -guerrilla  study  used  a  scenario  in  which  the 
hostile  guerrilla  farces  retreat  to  a  mountain  redoubt  to  be  surrounded 
by  US  troops  air-landed  by  helicopters.  This  scenario  makes  conventional 
tactics  palatable  in  counter-guerrilla  warfare ,  but  is  hardly  realistic. 

A  capable  guerrilla  leader  should  not  be  expected  to  use  such  disastrous 
tactics.  Adaptation  of  enemy  tactics  (e.g.,  rapid  dispersal)  in  face  of 
the  new  US  capability  for  air  landing  is  certainly  feasible.  A  com¬ 
parable  adaptation  to  the  enemy  capabilities  was  illustrated  during 
World  War  II.  German  air  defense  analyses  prior  to  that  war  were  based 
on  the  attacking  aircraft  using  certain  altitudes  that  were  optimum  for 
the  air  defense  batteries.  Allied  bomber  aircraft  did  not  oblige  and 
avoided  the  "optimum**  altitude  range. 

Some  studies  assume  maximum  future  enemy  capability  in  all  weapon 
areas.  The  enemy  cannot  simultaneously  maximize  all  of  this  capabilities 
if  constraints  in  physical  resources  and  budgets  are  present,  particu¬ 
larly  in  the  case  of  peacetime  budgets.  If  he  maximizes  his  strategic 
farces,  he  will  have  to  limit  his  tactical  capabilities,  and  vice  versa. 
Alternatives,  where  appropriate,  should  be  pitted  against  a  variety  of 
enemy  postures  and  the  choice  should  make  none  of  these  postures  parti¬ 
cularly  attractive  to  the  enemy. 

In  theory,  the  enemy  can  counter  every  system  we  design  and  our 
effectiveness  will  not  be  sufficiently  high  to  warrant  a  positive  de¬ 
cision.  The  real  question  is:  how  much  does  it  cost  the  enemy  in  time 
and  resources  to  effect  a  direct  counter?  If  the  price  is  very  high  he 
will  probably  seek  other  lesser  alternatives.  (See  question  6.) 


50.  IS  THE  EFFECTIVENESS  MEASURED  BY  ANALYSIS  OF  WARGAMES? 

When  effectiveness  is  measured  by  analysis  of  wargames  the  reviewer 
should  look  to  sensitivity  analysis  of  the  results.  As  a  rule,  wargames 
are  a  questionable  means  for  measuring  effectiveness  because  of  the 
difficulty  of  testing  the  sensitivity  of  the  results.  To  do  so  means 
challenging  the  effect  of  changes  in  players,  referees,  communications , 
as  well  as  payoff  functions.  (See  question  18.) 


51.  IS  THE  EVALUATION  OF  EFFECTIVENESS  BASED  ON  STRAIGHT  EXTRAPOLATION? 

Occasionally  a  study  may  evaluate  effectiveness  by  straight  (linear) 
extrapolation  from  the  measurement  of  effectiveness  of  a  small  unit. 

For  example,  a  hypothetical  study  may  shov  that  6  riflemen  can  destroy 
10  targets.  An  extrapolation  that  states  j.00  targets  can  be  destroyed 
by  60  riflemen  *s  not  justified  without  supporting  evidence.  The 
variables  in  target  and  fire  distribution  are  necessarily  the  seme 
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in  both  cases.  Further,  in  a  force  of  60  riflemen  the  percentage  who 
will  actually  fire  at  targets  may  not  be  the  same  as  for  a  force  of  6 
riflemen. 

Another  error  in  straight  or  linear  extrapolation  is  disregard  of 
the  element  of  diminishing  returns  or  marginal  utility.  For  example, 

200  missiles  do  rot  signify  twice  as  much  effectiveness  as  100  missiles 
if  there  are  wily  59  targets.  Furthermore,  sill  targets  are  not  of  equal 
value  or  importance. 


52.  ARE  THE  OPERATIONS  OF  OTHER  SERVICES  IGNORED? 

In  measuring  the  effectiveness  of  a  system  or  organisation,  consid¬ 
eration  must  be  given  to  the  operations  of  other  Services,  where  appro¬ 
priate.  Failure  to  do  so  is  the  equivalent  of  making  the  erroneous  Un¬ 
stated  assumption  that  only  the  Army  will  participate  in  the  operation. 
For  example,  the  measurement  of  effectiveness  of  Army  air  defense  oper¬ 
ations  must  consider  the  ccsnunicatlons,  camnand  and  control,  and  IFF 
aspects  of  operations  with  the  U.S.  Air  Force  and  allied  air  forces. 
Further,  the  effectiveness  of  certain  Army  operations  is  dependent  upon 
the  degree  of  air  superiority  achieved  by  the  Air  Force.  The  ability 
to  achieve  this  air  superiority  and  the  degree  of  dependence  upon  it 
should  be  examined.  (See  question  10.) 
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In  measuring  the  effectiveness  of  a  system  or  organisation,  the 
effects  on  other  Army  operations  should  be  considered.  For  example, 
the  use  of  tactical  nuclear  weapons  in  a  certain  manner  may  accomplish 
its  function  by  stopping  enemy  ground  movement.  However,  the  judgment 
of  the  effectiveness  of  the  system  should  also  examine  the  effect  chi 
the  ground  movement  of  U.  S.  units.  In  the  same  manner  certain  protec¬ 
tive  clothing  may  be  effective  against  enemy  chemical  agents.  However, 
the  clothing  may  cause  such  body  heating  that  it  can  only  be  worn  for 
very  short  periods. 


5U,  ARE  SOME  ASPECTS  OF  EFFECTIVENESS  INCOMMENSURABLE  OR  UNMEASURABLE  i 

The  reviewer  should  examine  carefully  the  treatment  of  incowaensur- 
ables  and  immeasurable  aspects  of  performance  in  the  total  measurement 
of  effectiveness.  Misleading  measures  of  effectiveness  are  now  often 
obtained  by  quantifying  such  aspects  as  morale,  or  leadership.  At 
times,  the  only  practicable  solution  may  be  a  qualitative  discussion  of 
the«e  factors. 


ui 


55..  DOBS  THE  EFFECTIVENESS  OF  A  FUTURE  SYSTB4  TAKE  INTO  ACCOUNT  THE 

TIMS  DIMERSICK? 

The  effectiveness  of  proposed  f  oture  systems  is  often  dependent 
upon  when  they  can  be  available  for  operational  use  and  the  total  opera- 
tional  life  span  of  the  systems.  In  examining  the  effect  of  the  time 
dimension  upon  effectiveness)  particular  attention  should  be  given  to 
(l)  the  time  between  the  present  and  the  initial  operational  avail¬ 
ability  of  the  complete  system,  and  (2)  the  latter  part  of  the  system 
operational  life  span. 

For  example,  the  effectiveness  of  Weapon  Y,  deployable  in  1972,  is 
compared  with  that  of  the  current  Weapon  X.  Weapon  Y  is  judged  to  be 
more  effective  and  requires  entirely  new  support  equipment  not  compatible 
with  that  of  Weapon  X.  This  equipment  cannot  be  operationally  avail¬ 
able  until  1971*.  It  is  very  possible  that  the  changeover  from  X  to  Y 
Implies  a  dip  in  effectiveness  during  the  1 970-7^  interval.  The  old 
weapon  is  becoming  obsolete  and  the  new  one  is  not  fully  effective.  A 
quick  fix  means  may  be  needed  to  bridge  this  gap  and  must  be  charged 
to  the  cost  of  X  and  Y. 

In  another  case  involving  the  time  dimension,  Weapon  B,  deployable 
in  1972,  replaces  Weapon  A  and  is  designed  to  perform  the  same  mission 
more  effectively.  Tt  is  stated  to  have  an  operational  life  of  15  years. 
Effectiveness  is  calculated  on  the  basis  of  the  1972  environment .  In 
the  1972  to  198?  period  (the  operational  life  of  B)  the  international 
environment,  and  hence  the  missions  may  undergo  major  changes.  In  fact, 
the  mission  for  which  A  is  designed  may  already  be  on  the  decli;.%. 
Effectiveness  is  not  always  constant  but  often  must  be  related  to  time. 

It  is  necessary  to  recognize  that  missions  do  not  remain  fixed. 
Effectiveness  should  not  be  evaluated  on  the  basis  of  either  a  specific 
probability  of  the  continuity  of  the  mission  or  of  a  specified  new  mis¬ 
sion.  Rather,  the  system  should  be  judged  on  its  ability  to  adjust  to 
such  changes. 

Similar  comments  apply  with  respect  to  changes  in  technology. 
Breakthrought s  cannot  be  predicted  very  successfully.  Nevertheless, 
certain  trends  are  noticeable.  For  example,  anti-tank  weapons  have 
improved  more  rapidly  than  ,anks  since  World  War  II.  The  sensitivity 
of  the  system  to  jumps  in  technology  is  a  vital  input  to  effectiveness 
evaluation  of  massive  long  .  ifetc-ae  systems. 


56.  ARE  EXPECTED  AND  AVERAGE  VALUES  USED  INCORRECTLY  TO  MEASURE 
EFFECTIVENESS? 

It  Is  an  error  to  employ  an  expected  value  or  average  as  part  of 
a  measure  of  effective-. ess  if  the  objective  really  requires  a  specified 
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minimal.  In  such  a  case,  the  possible  variances,  or  dispersions  about 
the  average,  constitute  an  unacceptable  risk  for  any  single  event. 

This  risk  is  unacceptable  even  though  o *rer  many  events  the  results  will 
average  out  to  the  expected  value. 

For  example,  assume  that  at  *he  same  cost,  air  defence  System  A 
destroys  from  D  to  99  of  100  approaching  enemy  aircraft  but  on  the 
average  destroys  ?0.  Syst  m  B.  on  the  other  hand,  destroys  from  25  to 
35  of  100  approaching  aircraft  with  an  average  destruction  of  30.  The 
risk  associated  with  the  possibility  that,  in  any  given  individual 
attack  by  100  aircraft.  System  A  may  not  destroy  any  aircraft  at  all, 
whereas  System  B  can  be  counted  on  to  destroy  at  least  25  aircraft, 
makes  A  an  unacceptable  system,  if  the  objective  is  air  defense.  If 
the  objective  were  destruction  of  as  many  enemy  aircraft  as  possible 
over  some  period  of  time  but  without  regard  to  their  damage  to  us,  (an 
uiuikely  objective)  System  A  would  be  preferred. 


57.  IF  QUANTITATIVE  MEASURES  OF  EFFECTIVENESS  ARE  UNATTAINABLE,  IS  A 

QUALITATIVE  COMPARISON  FEASIBLE? 

There  are  times  when  the  effectiveness  of  a  system  or  organization 
cannot  be  presented  adequately  in  quantitative  terms.  This  situation 
is  cannon  in  comparison  of  general  purpose  forces  such  as  in  studies  of 
alternative  divisions.  A  study  that  assigns  numerical  values  to  effec¬ 
tiveness  of  general  purpose  forces  should  be  examined  carefully. 

One  study  compared  alternative  divisions  in  terms  of  numerical 
scores.  E:h  of  the  six  basic  factors  (firepower,  intelligence,  mobil¬ 
ity,  conmand/control/coffimunications,  logistics,  survivability)  was 
given  a  numerical  value  and  these  values  were  summed  for  each  alter¬ 
native.  The  resulting  sums  were  compared  as  effectiveness  measures. 
These  numerical  values  are  likely  to  be  meaningless  because  he  six 
basic  factors  are  inputs  and  not  objectives.  They  combine  in  undeter¬ 
mined  ways  to  make  up  the  effectiveness  of  tactics.  The  tactic',  in 
turn,  combine  to  evolv**  strategies.  For  example,  deception  iact^'S 
strongly  involve  the  basic  Building  blocks  of  intelligence,  command/ 
control/  'cramunications,  and  mobility.  However,  this  does  not  mean  that 
we  cam  simply  add  up  so-called  scores  cf  these  three  basic  factors  and 
thereupon  compare  the  deception  capability  of  various  alternatives. 

A  qualitative  comparison  is  possible,  however.  Vaurious  pertinent 
aspects  can  be  described  and  characterized  by  Myes-no"  or  "good-fair- 
poor.  H  A  tabular  Caspar  Ison  can  h**  useful  in  weeding  out  some  alter¬ 
natives.  It  may  justifiable  to  say  that  Alternative  A  is  more  effective 
than  B  (denoted  in  a  certain  characteristic,  even  if  i *  is  not  know*' 
whether  A  is  1$  Times  or  twice  as  effective  as  B.  If  it  can  be  deter- 
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A  a  A 

mined  that  =r  and  •=■  we  have  a  partial  ordering  =-rr  i.e.,  we  cannot  dis¬ 
tinguish  between  B  and  C  but  either  is  inferioirtd  A.  We  may  obtain  a 
grouping  as  follows: 

A 

B,C,D 

E 


Let  us  reconsider  the  example  of  the  deception  tactic.  Its  key  in¬ 
gredients  are  mobility,  command/control/communications,  and  intelligence. 
Suppose  we  know  that  Division  A  is  more  mobile  than  B,  therefore,  ^  . 

If  we  should  arrive  e.t  the  same  ranking  for  the  other  two  basic  factors, 
then  we  conclude  that  A  is  also  true  for  the  deception  tactic.  On  the 
other  hana,  it  may  be  ®that  —  for  mobility  and  for  intelligence.  Then 
no  statement  can  be  made  for^the  relative  ranging  of  A  and  B  for  the  de¬ 
ception  tactic. 


In  this  manner,  tactics  of  interest  can  be  investigated  and  valid 
partial  orderings  of  alternatives  obtained.  We  may  find  dominant  alter 
natives.  Suppose  ve  obtain: 

C command,  Control, 

Mobility  Intelligence  Communications 


E.A 

B,C,D 


E 

A&C 


D 


We  have  no-  arned  that  D  is  dominated  by  A  for  all  three  basic  ^actors, 
and  hence  xor  the  deception  function.  So  D  can  be  eliminated  if  all  al  • 
ternatives  have  equal  c^st.  It  should  not  be  assumed  that  rankings  of 
alternatives  with  respect  to  the  tactical  level  can  only  be  derived  by 
buildup  and  integiation  from  the  basic  level.  There  may  be  direct  quali¬ 
tative  comparisons  with  respect  to,  say,  deception  effectiveness  as  a 
result  of  wargames  or  field  exercises.  A  combination  of  both  buildup 
and  direct  approaches  would  probably  prove  most  fruitful. 


The  reviewer  should  recognize  that  while  cost-effectiveness  analysis 
is  performed  preferably  by  quantitative  analysis,  there  are  limits  to 
suboptimizing  or  idealizing  the  problem  to  make  it  amenable  to  quantita¬ 
tive  analysis.  When  carried  too  far,  the  quantitative  results  are  often 
only  of  academic  interest  and  offer  little  or  ro  help  to  the  decision 
maker . 


58 .  IS  THE  EFFECTIVENESS  SENSITIVE  TO  CHANGES  IN  ASSUMPTIONS? 

The  effectiveness  derived  in  most  studies  is  usually  dependent  to 
a  degree  on  the  assumptions  The  reviewer  should  isolate  the  degree  of 
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dependence  and  determine  if  it  is  acceptable.  Generally,  a  good  study 
will  isolate  this  dependence,  where  it  exists,  and  lay  out  the  degree 
cf  dependence  by  various  kinds  of  sensitivity  or  contingency  analysis. 
The  assumptions  that  most  commonly  influence  effectiveness  and  are  often 
not  subjected  to  contingency  analysis  concern  the  locale,  the  time  and 
level  of  warfare,  and  enemy  forces  and  tactics. 

A  slight  change  in  any  of  these  assumptions  may  produce  significant 
changes  in  the  effectiveness  measured.  For  example,  additions  of  a  new 
ECM  band  width  to  the  enemy's  capability  may  drastically  degrade  an 
otherwise  outstanding  U.S.  system.  (See  questions  5  and  6.) 
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59.  ARE  THE  CRITERIA  IDENTIFIED? 

The  criteria,  or  tests  of  preferredness,  are  the  basis  for  the  con¬ 
clusions  and  recommendations.  The  criteria  should  be  stated  specifically 
and  clearly.  If  this  is  not  the  case,  the  reviewer  should  attempt  to 
identify  the  criteria  from  the  material  contained  in  the  study.  When 
this  does  not  prove  possible,  consideration  should  be  given  to  having  the 
study  returned  for  further  clarification.  This  is  particularly  important 
if  the  study  is  also  to  be  reviewed  by  agencies  outside  the  Army. 


60.  ARE  THE  CRITERIA  CONSISTENT  WITH  HIGHER  ECHELON  OBJECTIVES? 

No  matter  what  the  concern  of  a  study,  the  subject  falls  into  a 
larger  framework.  For  example,  problems  of  air  defense  of  the  CONUS  are 
aspects  of  the  larger  problem  of  restricting  possible  damage  to  the  CONUS 
to  certain  levels.  The  design  of  artillery  systems  is  part  of  the  larger 
problem  of  design  of  land  battle  forces.  Therefore,  the  reviewer  must 
determine  if  the  criteria  used  in  a  study  are  consistent  with  higher  level 
objectives.  This  requires  good  military  judgment  and  the  necessity  to 
examine  the  larger  context  of  the  problem.  If  the  study  criteria  are  not 
consistent  with  the  objectives  at  the  higher  level  then  the  wrong  problem 
may  be  solved.  Overall  Army  objectives  are  contained  in  documents  such 
as  the  Basic  Army  Strategic  Estimate  (BASE),  Army  Force  Development  Plan 
(AFDP),  Army  Strategic  Plan  (ASP),  and  the  Combat  Developments  Objective 
Guide  (CDOG). 

An  example  of  incorrectly  chosen  criteria  is  illustrated  in  the  use 
of  mobility  as  the  sole  criterion  in  the  selection  among  different  organi¬ 
zations.  A  study  could  conceivably  demonstrate  that  organization  A  can 
be  made  more  mobile  than  organizations  B  and  C  with  a  lesser  expenditure 
of  resources.  Yet  A  may  not  be  the  preferred  organization  because  the 
mobility  was  achieved  by  degrading  other  factors  that  contribute  to  the 
higher  objective  of  efficient  control  of  conflict  -situations  (e.g.,  fire¬ 
power,  sus+ainability,  etc.). 

61.  ARE  THE  CRITERIA  TOO  GENERAL? 

Generalized  criteria  are  suspect.  For  example,  a  study  may  state 
that  the  criterion  is  "the  system  with  maximim  military  worth"  or  the 
"best  system."  Such  generalizations  are  meaningless  and  cannot  be  related 
to  the  analysis  as  can  a  good  criterion  such  as  "the  minimum  cost  of  main¬ 
taining  a  (specified]  level  of  transport  capability  over  a  (specified] 
time  span." 
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62.  ARE  THE  CRITERIA  CT/ERDETERMINED? 

Overdetermined  criteria  lead  to  erroneous  conclusions.  A  criterion 
that  states  "to  maximise  the  damage  to  the  enemy  while  at  the  same  time 
minimizing  the  cost  to  the  U.S."  or  "causing  the  maximum  amount  of  casual¬ 
ties  With  the  least  expenditure  of  ammunition"  suggests  that  something 
can  be  obtained  for  nothing.  It  is  impossible  to  maximize  gain  and  sim¬ 
ultaneously  minimize  cost.  It  is  not  possible  to  increase  effectiveness 
without  some  increase  in  resources  (cost).  The  minimum  cost  is  to  do 
nothing- -and  achieve  no  effectiveness.  Occasionally  it  may  turn  out  that 
system  A  is  both  more  effedtive  than  system  B  and  costs  less.  However, 
system  A  will  not  be  both  more  effedtive  and  cost  less  when  compared 
with  additional  alternatives.  The  danger  of  using  an  overdetermined  cri¬ 
terion,  such  as  the  one  described,  is  that  it  leads  to  invalid  compromise 
criteria  by  using  some  erroneous  constraint  on  Effectiveness  or  cost  in 
an  effort  to  make  an  impocsible  test  seem  feasible. 


63.  ARE  GOOD  CRITERIA  APPLIED  TO  THE  WRONG  HKJBLEMS? 

At  times  a  valid  criterion  for  one  element;,  of /'the  problem  in  incor¬ 
rectly  applied  to  the  total  problem.  For  example,  a  hypothetical  study 
involving  proposed  surveillance  aircraft  shows  that  aircraft  A  offers 
greater  mission  flexibility  than  aircraft  B  at  the  same  cost  and  is  there¬ 
fore  preferred.  In  this  case,  the  choice  of  aircraft  is  not  the  real 
problem.  The  subsystems  carried  by  the  airdraft  are  really  more  crucial. 
The  all-weather  sensor  effectiveness  and  avionics  cost  may  even  determine 
whether  there  should  even  be  an  aircraft  A  or  B. 


64.  IS  THE  ABSOLUTE  SIZE  OF  GAIN  OR  COST  IGNORED? 

If  the  absolute  size  of  the  cost  of  a  system  alternative,  or  the 
effectiveness  to  be  achieved  by  it,  is  not  given  or  is  incorrect,  the 
analysis  often  leads  to  wrong  conclusions  and  recommendations.  For  ex¬ 
ample,  cost-effectiveness  curves  for  two  hypothetical  system  alternatives 
is  given  below: 


10  50  100 

ENEMY  TARGETS  DESTROYED 
(EFFECTIVENESS) 
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In  this  situation)  at  low  levels  of  effectiveness)  alternative  A 
is  preferred  (up  to  about  70  enemy  targets  destroyed);  at  larger  levels 
of  effectiveness,  alternative  B  is  preferred  (from  about  70  to  110).  If 
the  capability  to  destroy  more  than  110  enemy  targets  is  to  be  achieved, 
then  neither  alternatives  A  or  B  is  preferred  or  even  acceptable.  The 
crucial  question  is  how  many  enemy  targets  are  reqiired  to  be  destroyed. 

If  the  number  of  enemy  targets  to  be  destroyed  or  cost  limits  are  not 
indicated,  there  is  no  real  basis  to  recommend  either  alternative  A  or  B, 
or  some  other  alternative. 

Either  the  study  should  be  based  on  an  absolute  size  of  gain  or  cost 
required  or  the  study  should  present  a  cost-effectiveness  curve  (or  points) 
from  which  decisions  can  be  made.  If  the  study  presents  a  cost-effective¬ 
ness  curve  as  shown  above,  the  envelope  (indicated  by  line  of  X's  in  the 
graph)  along  the  bottom  says,  "This  is  a  curve  which  gives  the  most  for 
the  resources  expended,  and  other  things  have  to  be  taken  into  consider¬ 
ation  at  higher  levels  to  determine  what  the  absolute  gain  (number  of 
targets  destroyed)  should  be  or  the  maximum  resources  (cost)  that  can  be 
made  available." 

At  times,  studies  ignore  absolute  size  of  gain  or  cost  and  use  effec¬ 
tiveness  -to-cost  ratios.  The  flaw  in  the  use  of  such  ratios  is  the  absence 
of  any  specified  level  of  effectiveness  required  or  resources  available  as 
discussed  above.  If  the  level  of  activity  is  fixed,  a  ratio  may  be  useful 
in  ranking  among  alternative  systems.  However,  the  effectiveness -to-cost 
ratio  criterion  is  often  applied  when  the  level  of  activity  is  not  fixed. 
For  example,  in  the  graph  above  alternative  A  destroys  10  enemy  targets 
for  $1;  million,  and  alternative  B  destroys  100  enemy,  tatgfcbsj  for  $25 
Mllion.  •  If  onl^r  this  informat  ion.  we  re  converted  to  effect  ive  ness  «to- . ^ 
cost  ratios,  alternative  A  would  have  a  ratio  of  10:1  and  alternative  B, 
4:1.  Which  is  the  preferred?  If  one  did  not  look  at  the  absolute  level 
of  effectiveness  required  to  achieve  the  military  task  but  only  at  the 
effectiveness -to-cost  ratio,  then  alternative  A  would  be  preferred.  The 
selection  of  alternative  A  on  this  basis  may  be  correct,  but  only  by 
coincidence  and  is  obviously  wrong  when  th^  system  must  be  capable  of 
destroying  more  than  70  targets. 

Until  the  absolute  level  (magnitude)  of  effectiveness  or  the  absolute 
level  (magnitude)  of  the  cost  is  specified  the  preferred  alternative  cannot 
be  determined.  The  effectiveness-to-cost  ratio  can  be  misleading  and,  at 
times,  a  dangerous  criterion. 


.*•  ■  ***%•••***  «y e**y«Hiwiw  »."--^»Tt 


■**:■  ■*  W-**  iiSWl*^  ’ 


CONCLUSIONS  AND  RECOMMENDATIONS 

65.  ARE  THE  CONCLUSIONS  AND  RECOMMENDATIONS  LOGICALLY  DERIVED  FROM  THE 

MATERIAL  CONTAINED  IN  THE  STUDY? 

The  conclusions  and  recommendations  should  be  derived  logically  from 
the  material  contained  in  the  study.  Some  studies,  unfortunately,  draw 
conclusions  based  on  previous  studies  and  materials  that  are  not  fully 
documented  within  the  study  (mention  in  a  bibliography  is  hardly  suffi« 
cient).  If  input  from  another  study  is  essential,  it  should  be  documented 
and  explained  in  detail.  This  requires,  at  least,  a  statement  as  to  val¬ 
idity,  scope  of  application  and  uncertair  ,y  which  is  associated  with  the 
particular  input. 

The  determination  of  whether  the  conclusions  and  recommendations 
follow  logically  from  the  material  in  the  study  is  a  matter  of  judgment 
by  the  reviewer.  In  making  this  judgment,  the  reviewer  should  consider 
whether  other  prudent  study  agencies  would  probably  arrive  at  substan¬ 
tially  the  same  conclusions  and  recommendations  given  only  the  material 
contained  in  the  study. 


66.  HAVE  ALL  THE  SIGNIFICANT  RAMIFICATIONS  BEEN  CONSIDERED  IN  ARRIVING 

AT  THE  CONCLUSIONS  AND  RECOMMENDATIONS  CONSIDERED? 

Sometimes  a  study  fails  to  consider  all  the  pertinent  ramifications 
in  arriving  at  the  conclusions  and  recommendations.  These  unconsidered 
ramifications  may  either  influence  the  validity  of  the  conclusions  and 
recommendations  of  the  study  or  the  decisions  to  be  made  as  a  result  of 
the  study.  These  ramifications  are  often  referred  to  as  "spillovers." 

For  example,  if  a  hypothetical  study  recommended  adoption  of  an  engine 
requiring  a  new  type  of  fuel,  the  Army  supply  system  to  include  supply, 
storage  and  transportation  operations  would  be  affected.  Spillover  effect, 
are  not  always  negative.  For  example,  the  adoption  of  dehydrated  rations 
to  achieve  greater  shelf -life  may  also  reduce  construction  and  transporta¬ 
tion  costs  because  of  the  smaller  unit  volume  of  dehydrated  food. 

Other  ramifications  that  are  sometimes  neglected  are  factors  that 
should  be  considered  jointly  with  the  problem  under  study.  At  times,  con¬ 
sideration  of  such  joint  decisions  could  affect  the  conclusions  and  recom¬ 
mendations  of  the  study.  For  example,  a  study  may  recommend  adoption  of 
a  new  weapon  system  to  fulfill  a  certain  function.  However,  the  study  may 
neglect  to  examine  the  maintenance  support  and  the  maintenance  units  that 
would  have  to  be  in  existence  concurrently  with  the  proposed  weapon  system. 
The  resources  required  to  organize  and  maintain  the  maintenance  system 
will  influence  decisions  on  the  proposed  weapon  system. 
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If  significant  ramifications  are  uncovered  that  are  not  adequately 
considered,  the  reviewer  should,  if  possible,  determine  the  effects  of 
these  ramifications  on  the  conclusions  and  recommendations.  (See  ques¬ 
tion  2). 


67.  ABE  THE  CONCLUSIONS  AND  RECOMMENDATIONS  REALLY  FEASIBLE  IN  THE  LIGHT 

OF  POLITICAL,  CULTURAL,  POLICY  OR  OTHER  CONSIDERATIONS? 

In  reviewing  the  conclusions  and  recommendations  of  a  study,  it  is 
necessary  to  be  cognizant  of  the  real  world  in  which  the  Army  must  ope¬ 
rate.  At  times  some  recommendations  of  a  study  may  appear  to  be  eminently 
feasible  from  a  strictly  economic  or  military  view,  but  really  are  not 
so  in  the  light  of  other  considerations  that  influence  military  operations. 
For  example,  a  particular  toxic  chemical  munitions  system  may  be  demon¬ 
strated  to  be  superior,  considering  cost  and  effectiveness,  to  a  high 
explosive  munitions  system  for  accomplishing  a  certain  function.  However, 
because  of  national  policies  on  employment  of  toxic  chemicals,  the 
adoption  of  the  high  explosive  munitions  system  may  be  the  only  feasible 
solution. 

The  reviewer  should  also  consider  the  impact  of  policies  that  may 
not  have  been  known  to  the  agency  that  prepared  the  study  or  were  pro¬ 
mulgated  too  late  to  influence  the  study. 


68.  ARE  THE  CONCLUSIONS  AND  RECOMMENDATIONS  RELATED  TO  THE  LIMITATIONS 

OF  THE  STUDY? 

In  evaluating  conclusions  and  recommendations,  the  reviewer  should 
bear  in  mind  the  limitations  of  the  study.  Studies,  as  a  rule,  have 
varying  degrees  of  limitations.  The  more  common  types  of  limitations 
include  inadequate  data  base,  criticality  of  assumptions,  criticality 
of  uncertainties  and  validity  of  cost  and  effectiveness  models.  While 
the  limitations  may  be  treated  within  the  study,  the  dependence  of  the 
conclusions  and  recommendations  on  the  limitations  is  sometimes  neglec¬ 
ted.  For  example,  the  study  conclusions  and  recommendations  may  depend 
upon  the  validity  of  particular  assumptions  but  this  relation  may  not 
be  pointed  out. 

It  may  be  advisable  for  the  reviewer  to  refresh  his  memory  on  the 
study  limitations,  particularly  when  the  study  is  voluminous,  before 
evaluating  the  conclusions  and  recommendations. 
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69.  DO  THE  CONCLUSIONS  AND  RECOMMENDATIONS  INDICATE  BIAS? 

Studies  sometimes  unwittingly  reflect  bias  because  of  parochial  or 
institutional  interests.  To  sadist  in  detecting  bias,  the  reviewer  should 
consider  the  relation  of  the  agency  that  prepared  the  study  and  the  effects 
of  the  implementation  of  the  study  recommendations.  If  such  implementa¬ 
tion  does  not  appear  to  further  what  are  generally  considered  to  be  the 
particular  interests  of  the  preparing  agency,  then  one  occasional  form  of 
bias  is  probably  not  present.  Another  test  for  bias  is  to  judge  whether 
substantially  the  same  conclusions  and  recommendations  would  be  reached, 
based  on  the  material  in  the  study,  by  another  study  agency.  Bias  is 
often  displayed  by  arbitrarily  excluding  certain  reasonable  alternatives, 
maximizing  selected  enemy  capabilities,  treating  significant  uncertainties 
as  assumptions,  and  in  selection  of  effectiveness  criteria. 

A  relatively  minor  form  of  bias  is  sometimes  found  in  the  use  of 
prejudicial  adjectives.  Unnecessarily  referring  to  all  Air  Force  fixed 
wing  aircraft  as  "long  take  off  and  landing"  aircraft  is  an  example. 

This  type  of  bias  may  be  prejudicial  to  the  interests  of  the  Army  when 
the  study  is  reviewed  by  non-Army  agencies. 


70.  ARE  THE  CONCLUSIONS  AND  RECOMMENDATIONS  BASED  ON  EXTERNAL  CONSIDERATIONS? 

At  times,  recommended  selections  among  alternatives  must  be  made  in 
the  face  of  great  uncertainty.  A  study  may  find  that  several  alternatives 
exhibit  similar  cost-effectiveness,  but  the  results  are  very  sensitive  to 
the  values  assigned  to  the  inputs.  In  this  situation  some  studies  arrive 
at  conclusions  and  recommendations  based  on  considerations  other  than 
those  studied.  In  other  words,  the  study  agency  is  stating  that  after 
having  made  the  analysis,  the  application  of  the  criteria  does  not  lead 
to  preference,  but  indifference,  among  the  alternatives  and  therefore  the 
issue  was  decided  on  the  basis  of  other  unstudied  criteria.  In  situations 
of  this  kind,  when  recommendation  of  an  alternative  is  necessary,  sensi¬ 
tivity  to  new  criteria  must  be  fully  studied. 


71.  ARE  THE  CONCLUSIONS  AND  RECOMMENDATIONS  BASED  ON  INSIGNIFICANT 

DIFFERENCES? 

At  times  a  study  will  present  one  alternative  as  having  the  highest 
value  of  effectiveness  of  the  measures  applied  to  all  alternatives.  The 
difference  in  effectiveness  among  the  "optimum"  alternative  and  the  other 
alternatives  should  be  examined.  If  the  differences  are  relatively  slight 
and  probably  no  greater  ihan  the  uncertainties  in  the  data,  then  other 
grounds  should  also  be  demonstrated  for  selecting  among  the  alternatives 
that  are  close  in  effectiveness. 
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72.  IP  PRIORITIES  ARE  LISTED,  ARE  THEY  STATED  MEANINGFULLY? 


Conclusions  and  recommendations  often  list  items  of  equipment  in 
order  of  priority  of  recommended  procurement  or  adoption.  The  use  of 
this  technique  without  explanation,  particularly  for  materiel,  lo  often 
poor  because  it  provides  no  basis  for  a  decision.  For  example,  a  study 
may  conclude  that  in  order  to  accomplish  certain  functions,  infantry 
units  should  be  equipped  with  specified  items  of  equipment  that  are  listed 
in  order  of  priority.  Assume  that  the  items  found  necessary  by  the  study 
for  infantry  units  to  accomplish  the  required  functions  are,  in  order  of 
priority: 

(1)  Seven  League  Boots 

(2)  Disintegrator  Ray  Pistol 

(3)  Universal  Viewing  Device 

(4)  Camouflage  Suit  {makes  the  wearer  invisible) 

Although  the  study  concluded  that  all  of  these  items  are  required, 
the  listing  of  priorities  without  any  quantitative  considerations  could 
have  any  of  these  meanings: 

a.  Buy  all  of  the  Seven  League  Boots  required.  Then,  as  resources 
are  available*  buy  all  of  the  Disintegrator  Ray  Pistols  required.  Contin¬ 
ue  down  the  *ist  of  priorities  in  this  manner  until  the  available  resources 
are  exhausted.  This  meaning  also  infers  that  even  though  all  4  items  are 
required,  the  Army  can  do  without  the  lower  priority  items  if  sufficient 
resources  are  not  available  to  procure  them  all.  For  example,  with  limited 
resources  it  is  better  to  have  all  Seven  League  Boots  and  none  of  the 
other  items  rather  than  some  of  each  item. 

b.  Buy  all  4  items  at  once  but  spend  more  money  on  Seven  League 
Boots  than  on  Disintegrator  Ray  Pistols  and  even  less  amounts  on  Universal 
Viewing  Devices  and  Camouflage  Suits. 

When  faced  with  this  kind  of  situation,  consideration  should  be 
given  to  returning  th^  study  to  the  preparing  agency  for  further  recom- 
wndetions  on  how  much  should  be  allocated  to  each  item  for  various  bud¬ 
get  levels,  either  given  or  assumed. 


73.  ARE  THE  CONCLUSIONS  AND  RECOMMENDATIONS  INTUITIVELY  SATISFYING? 

When  the  conclusions  and  recommendations  of  the  study  are  not  intui¬ 
tively  satisfying,  the  reviewer  should  attempt  to  isolate  the  cause.  If 
the  study  fills  tc  demonstrate  by  data,  models  and  other  means  that  the 
reviewer*#  intuition  was  wrong,  then  further  exauination  is  required  to 
determine  if  some  subtle  considerations  have  not  been  considered  because 
of  oversimplification  or  other  reasons  which  the  reviewer  intuitively 
knows  are  pertinent. 
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GLOSSARY 


Alternative 


One  of  several  different  ways  of  achieving  a  desired  capability. 


Building  Block  Cost 

One  kind  of  a  rough  estimate  of  the  cost  of  an  alternative  for  planning 
purposes.  The  estimate  is  not  time-phased  and  does  not  provide  for  varia¬ 
tions  such  as  in  the  manni.ig  of  the  unit  or  cost-quantity  relationships. 


Contingency  Analysis 

Repetition  of  an  analysis  with  different  qualitative  assumptions  such  as 
theater,  or  type  of  conflict,  to  determine  their  effects  on  the  results 
of  the  initial  analysis. 


Commensurability 

The  capability  of  two  qualities  or  values  to  be  measured  by  a  meaningful 
relevant  conmon  index.  For  example,  machine  guns  and  rifles  are  commensur¬ 
able  either  in  dollar  cost  or  in  effectiveness,  e.g.,  enemy  casualties. 
However,  machine  guns  and  friendly  casualties  are  not  commensurable  in 
terms  of  dollars. 


Cost  Analysis 

The  analysis  and  determination  of  coat  (total  resource  implications)  of 
interrelated  activities  and  equipments  to  determine  the  relative  costs  of 
alternative  systems,  organisations,  and  force  structures.  Cost  analysis 
results  are  not  designed  to  have  the  precision  required  for  budget  purposes. 


Cost-Effectiveness  Analysis 

The  quantitative  examination  of  alternative  prospective  systems  for  the 
purpose  of  identifying  the  preferred  system  and  its  associated  equipment, 
organizations,  etc.  The  examination  aims  at  finding  more  precise  answers 
to  a  question  and  not  at  justifying  a  conclusion.  The  analytical  process 
includes  trade-offs  among  alternatives,  design  of  additional  alternatives, 
and  the  measurement  of  the  effectiveness  and  cost  of  the  alternatives. 
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Cost  Estimate 


The  estimated  cost  of  &  component  or  aggregation  of  components.  The 
analysis  and  determination  of  cost  of  interrelated  activities  and  equip¬ 
ments  is  cost  analysis. 


Cost  Estimating  Relation  (CBR) 

A  numerical  expression  of  the  lini  between  a  physical  characteristic, 
resource  or  activity  and  a  particular  cost  associated  with  it;  e.g., 
cost  of  aircraft  maintenance  per  flying  hour. 


Cost  Model 


An  ordered  arrangement  of  data  and  equations  that  permits  translating 
physical  resources  into  costs. 


Criterion 

Test  of  preference.  For  example,  highest  effectiveness  (expecte'-’.  wartime 
capability)  at  the  same  net  future  peacetime  cost  of  ownership. 


Deterministic  Model 

A  model  that  permits  no  uncertainty  in  the  magnitudes  of  either  inputs 
or  outputs.  An  example  from  gunnery  is: 


W*  * 


RM 

1000  ’ 


where 


V  is  the  lateral  distance  at  range  R;  R  is  the  range  and  M  is  the  angular 
measure  in  mils  of  the  arc  subtended  by  W  at  range  R.  For  any  set  of 
given  values  for  R  and  M  there  is  one  and  only  on*  value  for  V.  Many 
deterministic  models  use  an  average  as  a  constant  value  input. 


Force  Structure  Analysis 

The  analysis  of  proposed  forces  to  obtain  a  picture  of  resource  implica¬ 
tions  for  planning. 


Force  Structure  Costing 


The  determination  of  the  resource  implications  (manpower,  material, 
support,  training,  etc.)  in  dollar  terms  of  a  given  force  structure  or 
change  to  it. 


Incocane  ns  urhbility 

The  lack  of  capability  of  two  qualities  or  valueo  to  be  Manured  by  a 
meaningful  relevant  caramon  index.  (See  Caramensurability) 


Incremental  Cost 


The  net  additional  costs  that  would  result  from  choosing  a  particular 
alternative  as  compared  to  a  continuation  of  the  present  piograa. 


Individual  System  (Organization)  Coating 

The  determination  of  the  total  resource  implications  of  a  system  (organ¬ 
ization)  without,  consideration  of  the  interaction  of  the  system  (organiz¬ 
ation)  as  part  of  a  force  structure. 


Investment  Cost 


The  cost  beyond  the  Research  and  Development  phase  to  introduce  a  .ev 
capability  into  operational  use. 


Learning  Curve 

The  cost-quantity  relationships  for  estimating  costs  of  equipment. 
Generally  used  to  predict  or  describe  the  decrease  in  the  cost  of  a  unit 
as  the  number  of  unit,  produced  Increases. 


Linear  Programming 

\  mathematical  method  'used  to  determine  the  best  use  of  limited  resources 
to  achieve  a  desired  result  when  the  limitations  on  the  resources  are 
expressed  by  simultaneous  linear  relations  (x  *  a  ♦  by).  Linear  program¬ 
ming  is  applicable  to  problems  involving  resource  allocations  and  schedul¬ 
ing.  For  example,  linear  programing  has  been  'used  to  determine  the  best 
scheduling  of  facilities  for  a  given  number  of  students  taking  courses 


requiring  different  Kinds  of  classrooms,  shops,  and  ranges  where  there  are 
limitations  on  the  number  of  instructors  and  on  the  numbers  and  sites  of 
classrooms,  shops,  ranges. 


Model 


Relations  used  to  portray  real  or  expected  conditions,  actions  or  effects 
in  ord_*r  to  predict  the  outcome  of  actions.  For  example,  a  road  space 
and  time  length  nomograph  (FM101-10,  page  131)  is  a  mathematical  and  graphic 
model.  It  portrays  real  conditions,  e.g.,  average  road  spaces  and  time 
lengths  under  certain  conditions.  It  is  use  to  predict  the  outcome  of 
an  action,  e.g.,  determination  of  road  space  and  time  length  and  so  forth 
resulting  from  the  movement  of  a  certain  force  under  certain  conditions. 

A  service  maneuver  could  be  considered  a  non-mat hematical  model.  It 
portrays  the  expected  world  (combat)  as  realistically  as  circumstances 
permit  in  an  effort  to  foresee,  even  to  a  limited  extent,  the  outcome  of 
certain  actions  (use  of  certain  organizations,  tactics,  and "equipment ) . 


Monte  Carlo  Technique 

Used  to  solve  probability  problems  by  employing  sampling  to  estimate  the 
answer  to  a  precise  mathematical  problem.  A  "game"  or  model  is  formulated 
so  that  the  average  of  the  results  of  a  large  number  of  plays  of  the  game 
or  model  is  approximately  the  number  being  estimated.  Individual  plays  of 
the  game  or  model  are  often  used  to  determine  the  probability  of  a  random 
event.  For  example,  in  the  study  of  the  probable  vulnerability  of  a  tar¬ 
get,  the  Monte  Carlo  technique  is  often  applied  to  the  error  of  the  attack¬ 
ing  weapon.  A  model  is  prepared  containing  random  numbers  with  the  same 
distribution  as  CEI  values.  A  selection  of  a  random  number  (random  event) 
gives  the  probable  CEP  for  the  simulated  firing  of  the  weapon.  (See 
question  42) 


Objective 

The  purpose  to  be  achieved  or  the  position  to  be  obtained.  Objectives 
vary  with  the  level  of  suboptimization  of  the  study. 


Operating  Cost 

The  recurring  cost  required  to  operate  and  maintain  an  operational  cap¬ 
ability. 
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Parameter 


An  element  of  a  problem  that  may  be  either  a  constant  or  a  variable.  For 
example,  the  demand  for  and  availability  of  parts  are  parameters  in  some 
logistics  problems. 

An  analysis  conducted  with  assumed  instead  of  expected  or  actual 
values.  In  ne  absence  of  data  from  experiments,  or  other  sources,  para¬ 
metric  analyses  are  used  to  examine  a  problem,  to  identify  sensitive 
parameters,  and  to  obtain  reasonable  approximations  of  final  results 
(upper  ana  lower  bounds).  In  a  parametric  analysis,  a  range  of  values 
for  each  parameter  are  assumed  which  will  bracket  the  expected  values  of 
that  parameter,  and  a  solution  to  the  problem  obtained  for  each  set  of 
assumed  parameter  values.  In  summary,  a  parametric  analysis  answers  the 
figurative  question:  "if  the  values  of  the  parameters  were  such  and  such, 
then  what  will  the  results  be?" 


Psacetime  Costs 


The  costs  associated  with  developing,  buying,  and  maintaining  a  capability 
for  potential  war  during  peacetime.  Included  in  these  peacetime  costs  are 
the  resource  implications  of  buying  and  maintaining  a  war  reserve. 


Probabilistic  Mode 1 


A  model  that  makes  allowances  for  randomness  in  one  or  more  of  the  factors 
that  determine  the  outputs  of  the  model.  For  example,  an.  inventory  model 
that  optimizes  an  inventory  policy  to  avoid  inventory  shortages  is  proba¬ 
bilistic  if  it  takes  explicit  account  of  uncertainty  over  time,  in  the 
distribution  of  demands  on  the  inventory.  On  the  other  hand,  the  model 
would  be  deterministic  if  it  assumed  that  che  rate  of  demand  against  the 
inventory  is  always  the  same  (usually  the  estimated  average  demand).  In 
this  example,  a  deterministic  model  would  most  probably  give  answers  that 
would  lead  to  bad  inventory  policies.  However,  there  are  times  when  the 
use  of  a  deterministic  model  in  a  probabilistic  situation  does  no  harm. 
(See  question  42) 


Queuing  Theory 

A  theory  that  deals  with  the  analysis  of  costs  and  effectiveness  when 
items  appear  with  some  randomness  for  processing  at  a  facility  with  a 
capacity  for  processing  simultaneously  fewer  items  than  may  be  waiting 
at  a  given  time.  The  costs  are  costs  of  waiting  and  of  providing  the 
capacity  to  reduce  the  amount  of  waiting.  Examples  of  queuing  problems 
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are:  (l)  determination  of  a  niaber  cf  checkout  counters  at  a  supermarket 
that  minimizes  the  sum  of  costs  of  customer  dij satisfaction  if  they  must 
wait  in  line  and  costs  of  providing  additional  checkers;  (2)  determination 
of  the  capacity  of  communication  systems  to  minimize  the  costs  of  commun¬ 
ications  capacity  and  of  delays  in  the  processing  of  messages. 


Research  and  Development  (R&D)  Costs 

The  cost  of  developing  a  new  capability  to  the  point  where  it  is  ready 
for  procurement  for  operational  ’units. 


Resource  Impact 


The  cost  of  adopting  a  course  of  action  stated  in  measurable  terms. 
Resource  impacts  cannot  always  be  reduced  to  dollar  terms. 


Risk 

As  used  in  cost-effectiveness  analysis  and  operations  research,  a  situa¬ 
tion  is  characterized  as  risk  if  it  is  possible  to  aesr  Ibe  all  possible 
outcomes  and  to  assign  meaningful  objective  numerical  probability  weights 
to  Bach  one.  For  example,  an  action  might  lead  to  this  risky  outcome:  a 
reward  of  $10  if  a  "fair"  coin  comes  up  heads,  and  a  loss  of  $5  if  it 
comes  up  tails.  Another  example,  50$  of  all  missiles  fired  can  be  expec¬ 
ted  to  land  within  one  CEP  of  the  target. 


Sensitivity  Analysis 

Repetition  of  an  analysis  with  different  quantitative  values  for  cost  or 
operational  assumptions  or  estimates  such  as  hit -kill  probabilities, 
activity  rates,  or  R&D  costs,  in  order  to  determine  their  effects  for  the 
purposes  of  comparison  with  the  results  of  the  basic  analysis.  If  a  small 
change  in  an  assumption  results  in  a  proportionately  or  greater  change  in 
the  results,  then  the  results  are  said  to  be  sensitive  to  that  assumption 
or  parameter. 


Spillover 

The  effects  of  one  system  or  organization  upon  another  system  or  organ¬ 
ization. 


i 

I 
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Stochastic  Process 


The  statistical  concept  underlying  the  prediction  of  the  condition  of  an 
element  of  a  larger  group  when  the  probable  average  condition  of  the 
larger  group  is  known.  For  example,  assume  that  an  armored  division, 
under  certain  circumstances,  has  on  the  average  a  certain  number  of  tanks 
deadlined  for  unscheduled  maintenance.  The  probability  that  any  given  tank 
under  the  same  circumstances,  will  be  deadlined  for  unscheduled  maintenance 
on  a  specific  day  is  described  by  a  stochastic  process. 


Suboptimization 

Optimization  refers  to  a  selection  of  a  set  of  actions  that  maximize  the 
achievement  of  sane  objective  subject  to  all  of  the  real  constraints  that 
exist.  For  example,  one  may  optimize  a  choice  of  weapons  for  achieving 
certain;  objectives  of  a  decision  but  within  the  given  constraint  of  a 
certain  maximum  cost  of  a  division.  But  one  suboptimizes  on  achievement 
of  the  division  objective  if  he  is  given  discretion  only  over  the  amount 
and  kind  of  armor  and  is  given  a  maximum  umount  of  money  to  spend  on  armcr. 
The  objective  he  maximizes  directly  may  be  only  the  mission  of  armor  in 
the  division's  objective.  Such  a  suboptimization  will  yield  something 
inferior  to  an  optimized  expenditure  on  different  kinds  of  armor  if  the 
total  budget  for  armor  given  to  the  suboptimizer  is  really  not  optimal, 
or  if  there  are  interdependencies  between  decisions  on  armor  and  decisions 
on  other  things  that  are  outside  the  discretion  of  the  person  suboptimiz¬ 
ing  on  armor. 


Sunk  Cost 


The  costs  expended  in  the  past  and  which  ar '  now  irrelevant  to  present 
and  future  decisions. 


System 

An  integrated  relationship  of  men,  equipment  and  methods  appropriately 
organized  to  accomplish  defined  tasks. 


System  Analysis 

There  is  no  commonly  accepted  definition  for  this  term.  It  is  often 
used  synonymously  with  cost-effectiveness  analysis.  Other  definitions 
used  in  engineering,  management,  and  computer  system  design  are: 
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a.  (Engineering)  ftirt  of  the  preliminary  design  lead¬ 
ing  to  selection  of  preferred  design  approaches. 

b.  (Management)  Study  of  methods  for  obtaining  desired 
results.  Emphasizes  reporting  and  control  tech* 
niqueb . 

c.  (Computers)  Part  of  the  design  of  systems  of  com¬ 
puters  and  peripheral  equipment  for  miscellaneous 
uses. 


Time-Phased  Cost 

A  presentation  of  the  cost  results  broken  down  by  the  time  period  in 
which  the  costs  occur  rather  than  a  single  total  cost  figure. 


Total  Obligation  Authority  (TQA) 

The  cost  allocated  to  a  given  system  or  organization.  This  cost  when 
related  tor,a  specific  time  period,  for  example  a  year,  represents  obli¬ 
gations  that  can  be  incurred  during  that  year  and  not  necessarily  expend¬ 
itures.  The  total  obligation  authority  for  a  specific  year  to  furnish  a 
house  is  the  cqfit  of  what  can  be  ordered  during  that  year  even  if  deliver¬ 
ies  and  payments  are  made  in  later  years. 


Uncertainty 

A  situation  is  uncertain  if  there  is  no  objective  basis  for  assigning 
numerical  probability  weights  to  the  different  possible  outcomes  or  there 
is  no  way  to  describe  the  possible  outcomes.  For  example,  the  probability 
of  a  foreign  nation  continuing  to  furnish  the  U.S.  with  base  rights  is  an 
uncertainty. 


Wartime  Costs 


Those  costs  associated  with  producing  and  delivering  resources  to  the 
theater  after  the  war  has  started. 


« 
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SELECTED  QUESTION 


1.  Is  the  problem  stated  the  real  problem? 

4.  Are  the  assumptions  unduly  restrictive? 

12.  Are  any  feasible  and  significant  alternatives  omitted? 

13.  Is  the  study  adequately  documented? 

14.  Are  the  facts  stated  correct? 

22.  Are  the  cost  estimates  relevant? 

23.  Are  incremental  costs  considered? 

31.  Is  an  amortized  cost  used? 

37.  Are  the  models  adequately  identified  and  explained? 

44.  Are  the  models  intuitively  acceptable? 

;,6.  Is  the  effectiveness  measure  appropriate  to  the 
function  or  mission? 

57.  If  quantitative  measures  of  effectiveness  are 

unattainable,  is  a  qualitative  comparison  feasible? 

60.  Are  the  criteria  consistent  with  higher  echelon 
objectives '( 

66.  Have  all  the  significant  ramifications  been 
considered  in  arriving  at  the  conclusions  and 
recommendations  considered? 

73*  Are  the  conclusions  and  recommendations  intuitively 
satisfying? 
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A  written  version  of  a  lecture  given  at  the  Industrial  College 
of  th*  Armed  Forces.  Presents  a  discussion  on  the  use  of  economic 
analysis  as  a  conceptual  framework  in  assisting  military  planners  and 
decision-makers.  It  uses  the  programmed  learninte  technique  in  which 
the  reader  is  introduced  to  cost-effectiveness  analysis  through  a 
number  of  short  sequential  steps. 

4.  Hayes,  James  H. ,  "Systems  Analysis,"  Military  Review  (April  1965 ) 

A  general  discussion  on  the  structuring  of  cost-effectiveness 
analysis.  Good  discussion  on  the  principles  of  construction  and  use 
of  models. 

5.  Heymont,  Irving,  "The  Role  of  the  $  in  Military  Planning,"  Army 
(March  1965) 

A  general  dfecussion  on  the  economic  factor  in  military  planning 
with  emphasis  on  relating  qualitative  and  quantitative  requirements. 
Several  techniques  for  dealing  with  quantitative  requirements  are 
discussed. 

6.  Hitch,  0.  J.,  "Operations  Analysis  in  the  Department  of  Defense," 
Management  Views  (1963-1964)  Vol.  DC,  Part  2,  Selected  Speeches, 
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U.S.  Army  Management  School.  (Reproduced  in  Item  1  above) 

The  Assistant  Secretary  of  Defense  (Comptroller)  discusses  the 
use  of  quantitative  analysis  of  the  cost  and  effectiveness  to  assist 
in  selecting  the  preferred  alternative  from  smong  various  courses  of 
action.  A  major  portion  of  this  article  is  devoted  to  what  cost-effec¬ 
tiveness  analysis  is  not. 

7.  Hitch,  C.  J.  and  McKean,  R.  N. ,  The  Economics  of  Defense  in  the 
Nuclear  Age,  Report  R-346,  The  RAND  Corporation  (March  I960) 

Harvard  University  Press,  Cambridge,  Massachusetts,  I960,  420  pp. 

(AD  243098) 

This  book  is  considered  one  of  the  definitive  works  on  cost- 
effectiveness  analysis.  Contains  a  number  of  illustrative  examples, 
most  of  which  concern  Air  Force  problems.  The  book  is  divided  into 
three  sections:  (l)  Part  I:  The  Resources  Available  for  Defense; 

(2)  Part  II:  Efficiency  in  Using  Defense  Resources;  (3)  Part  III: 
Special  Problems  and  Applications.  Part  II  is  particularly  recommended. 

8.  Kahn,  H.  and  Mann,  I.,  "Ten  Common  Pitfalls,1'  RM-1937  (-July  1957), 

The  RAND  Corporation  (AD  133035) 

Identifies  and  discusses  ten  common  mistakes  found  in  doing 
systems  analysis.  Written  for  both  the  user  and  preparer  of  cost- 
effectiveness  analyses. 

9.  Large,  J.  P. ,  Ed.,  Concepts  and  Procedures  of  Cost  Analysis, 

RM-3589-PR  (June  1963),  The  RAND  Corporation  ( AD  111 5^) 

Presents  the  basic  concepts  of  cost  analysis  and  certain 
specialized  procedures  for  estimating  the  cost  of  resources.  While 
the  examples  are  Air  Force,  the  basic  cost  analysis  approach  is  appli¬ 
cable  to  other  services.  This  basic  approach  is  utilized  by  06D  in 
guidance  for  Program  Change  Proposals  and  Program  Definition  Rtaae. 

10.  Linstone,  H.  A.,  The  Weapons  Planning  Program  for  General  Rurposc 
Forces:  A  Suggested  Approach.  RM-3202-ISA  (April  19^2),  The  RAND 
Corporation,  (AD  4l4$17) 

Does  not  specifically  discuss  cost-effectiveness  analysis;  how¬ 
ever,  it  does  present  a  suggested  approach  at  determining  projected 
force  structures  for  general  purpose  forces.  Presents  a  suggested 
technique  to  relate  equipment  to  operational  needs  sc  that  the  enuaer- 
ation  of  hardware  alternatives  fo\  general  purpose  forces  is  facilita¬ 
ted. 
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11.  Murray,  R.,  2d,  "Some  Facets  of  Systems  Analysis,"  Management  Views 
(1963-1964)  Vol.  DC,  Part  2,  Selected  Speeches,  U.S.  Army  Management 
School . 

t 

The  Director  of  Systems  Analysis  in  the  Office  of  the  Deputy 
Assistant  Secretary  of  Defense  (SA)  describes  a  simplified  example 
of  a  systems  analysis  problem  with  special  emphasis  on  the  use  of 
sunk  costs  and  the  dangers  of  using  an  effectiveness-to-cost  ratios. 

12.  Quade,  E.  S.,  Ed.  Analysis  for  Military  Decisions,  Rand -McNally. 

The  written  version  of  a  five-day  course  entitled  "An  Apprecia¬ 
tion  of  Analysis  for  Military  Decisions"  which  was  given  to  military 
officers  and  civilians  associated  with  the  armed  forces.  The  examples 
presented  are  Air  Force  oriented.  The  book  states  that  it  is  little 
more  than  a  critical  evaluation  of  systems  analysis :  however,  the  con¬ 
cepts  and  methodology  discussed  should  be  useful  to  a  reviewer  of 
cost-effectiveness  analyses. 

13.  Quade,  E.  S.,  Military  Systems  Analysis.  RM-3452-PR  (January  .1963), 

The  RAND  Corporation  (AD  292026) 

The  publication  attempts  to  survey  the  problems  and  procedures 
of  military  cost-effectiveness  analysis.  A  good  introduction  to 
systems  analysis. 

NOTE:  A  number  of  publications  listed  above  are  available  through  the  Defense 
Documentation  Center  (DDC).  The  DDC  will  send  single  copies  of  reques¬ 
ted  publications  to  government  agencies  and  to  non-government  organiza¬ 
tions  having  Department  of  Defense  contracts  and  an  approved  Field-of- 
Interest  Register  on  file  at  DDC.  Publications  should  be  ordered  by 
DDC  Document  (AD  or  ATI)  numbers  on  DDC  Form  I  Complete  instructions 
and  the  forms  necessary  for  requesting  documents  are  contained  in 
"DDC  Service  Information  and  Forms"  which  may  be  obtained  from: 

Defense  Documentation  Center 
Cameron  Station 
Alexandria,  Virginia,  22314 
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